Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250109 20:52:38.998884 416 test_util.cc:274] Using random seed: 681226048
2025-01-09T20:52:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-09T20:52:39Z Disabled control of system clock
I20250109 20:52:39.027104 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37941
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250109 20:52:39.333966 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:52:39.334233 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:52:39.334348 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:52:39.341624 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:52:39.341753 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:52:39.341817 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:52:39.341885 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:52:39.351047 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:52:39.352557 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:52:39.354280 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:52:39.365319 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:39.366494 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:39.367470 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:39.368592 424 server_base.cc:1034] running on GCE node
I20250109 20:52:39.369375 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:52:39.370673 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:52:39.371872 424 hybrid_clock.cc:648] HybridClock initialized: now 1736455959371785 us; error 67 us; skew 500 ppm
I20250109 20:52:39.372167 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:52:39.379163 424 webserver.cc:458] Webserver started at http://127.0.104.62:34241/ using document root <none> and password file <none>
I20250109 20:52:39.379892 424 fs_manager.cc:362] Metadata directory not provided
I20250109 20:52:39.380008 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:52:39.380285 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:52:39.382848 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/data/instance:
uuid: "8328960890b142248073cec450e957d8"
format_stamp: "Formatted at 2025-01-09 20:52:39 on dist-test-slave-612g"
I20250109 20:52:39.383553 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/wal/instance:
uuid: "8328960890b142248073cec450e957d8"
format_stamp: "Formatted at 2025-01-09 20:52:39 on dist-test-slave-612g"
I20250109 20:52:39.388398 424 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20250109 20:52:39.391582 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:39.392812 424 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:52:39.392961 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/wal
uuid: "8328960890b142248073cec450e957d8"
format_stamp: "Formatted at 2025-01-09 20:52:39 on dist-test-slave-612g"
I20250109 20:52:39.393134 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:52:39.422133 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:52:39.422950 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:52:39.423231 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:52:39.441246 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37941
I20250109 20:52:39.441275 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37941 every 8 connection(s)
I20250109 20:52:39.442430 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/data/info.pb
I20250109 20:52:39.445868 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:39.450886 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu as pid 424
I20250109 20:52:39.451131 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-0/wal/instance
I20250109 20:52:39.451591 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:52:39.453445 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39231
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:52:39.466874 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:52:39.467862 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39231: connect: Connection refused (error 111)
W20250109 20:52:39.469259 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39231: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39231: connect: Connection refused (error 111)
I20250109 20:52:39.518671 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } attempt: 1
W20250109 20:52:39.520364 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39231: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39231: connect: Connection refused (error 111)
I20250109 20:52:39.588660 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } attempt: 2
W20250109 20:52:39.590409 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39231: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39231: connect: Connection refused (error 111)
I20250109 20:52:39.681687 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } attempt: 3
W20250109 20:52:39.683399 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39231: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39231: connect: Connection refused (error 111)
W20250109 20:52:39.751245 494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:52:39.751611 494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:52:39.751739 494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:52:39.759850 494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:52:39.760398 494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:52:39.760459 494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:52:39.760526 494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:52:39.768433 494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:39231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:52:39.769893 494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:52:39.771344 494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:52:39.781311 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:39.782341 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:39.783308 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:39.784327 494 server_base.cc:1034] running on GCE node
I20250109 20:52:39.785025 494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:52:39.786355 494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:52:39.787565 494 hybrid_clock.cc:648] HybridClock initialized: now 1736455959787506 us; error 43 us; skew 500 ppm
I20250109 20:52:39.787871 494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:52:39.790014 494 webserver.cc:458] Webserver started at http://127.0.104.61:32779/ using document root <none> and password file <none>
I20250109 20:52:39.790669 494 fs_manager.cc:362] Metadata directory not provided
I20250109 20:52:39.790778 494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:52:39.791042 494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:52:39.793920 494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/data/instance:
uuid: "0f066b28f8dd4e9cae5c64bd93feeb51"
format_stamp: "Formatted at 2025-01-09 20:52:39 on dist-test-slave-612g"
I20250109 20:52:39.794616 494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/wal/instance:
uuid: "0f066b28f8dd4e9cae5c64bd93feeb51"
format_stamp: "Formatted at 2025-01-09 20:52:39 on dist-test-slave-612g"
I20250109 20:52:39.799232 494 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250109 20:52:39.802096 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:39.803233 494 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:39.803380 494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/wal
uuid: "0f066b28f8dd4e9cae5c64bd93feeb51"
format_stamp: "Formatted at 2025-01-09 20:52:39 on dist-test-slave-612g"
I20250109 20:52:39.803544 494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:52:39.819141 494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:52:39.819840 494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:52:39.820084 494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:52:39.826660 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } attempt: 4
W20250109 20:52:39.828585 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:39231: Network error: Client connection negotiation failed: client connection to 127.0.104.61:39231: connect: Connection refused (error 111)
I20250109 20:52:39.836431 494 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:39231
I20250109 20:52:39.836457 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:39231 every 8 connection(s)
I20250109 20:52:39.837608 494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/data/info.pb
I20250109 20:52:39.841420 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:39.846351 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:52:39.847476 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu as pid 494
I20250109 20:52:39.847615 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-1/wal/instance
I20250109 20:52:39.848670 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41583
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:52:39.858534 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:52:39.867511 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:52:39.868525 511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111)
W20250109 20:52:39.869880 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:41583: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111)
I20250109 20:52:39.919193 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } attempt: 1
W20250109 20:52:39.920898 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:41583: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111)
I20250109 20:52:39.989099 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } attempt: 2
W20250109 20:52:39.990947 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:41583: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111)
I20250109 20:52:40.082159 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } attempt: 3
W20250109 20:52:40.084076 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:41583: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111)
I20250109 20:52:40.127831 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } attempt: 5
I20250109 20:52:40.134299 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
W20250109 20:52:40.135763 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:41583: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111)
W20250109 20:52:40.151290 564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:52:40.151539 564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:52:40.151607 564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:52:40.158380 564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:52:40.158485 564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:52:40.158528 564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:52:40.158577 564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:52:40.167261 564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:52:40.168747 564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:52:40.170271 564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:52:40.180648 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:40.181378 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:40.182557 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:40.183655 564 server_base.cc:1034] running on GCE node
I20250109 20:52:40.184407 564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:52:40.185669 564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:52:40.186862 564 hybrid_clock.cc:648] HybridClock initialized: now 1736455960186813 us; error 36 us; skew 500 ppm
I20250109 20:52:40.186957 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } attempt: 1
I20250109 20:52:40.187157 564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250109 20:52:40.188385 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:41583: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111)
I20250109 20:52:40.189337 564 webserver.cc:458] Webserver started at http://127.0.104.60:46069/ using document root <none> and password file <none>
I20250109 20:52:40.190003 564 fs_manager.cc:362] Metadata directory not provided
I20250109 20:52:40.190093 564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:52:40.190315 564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:52:40.192642 564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/instance:
uuid: "36a29ab2002d4eb389cd11778988e1e9"
format_stamp: "Formatted at 2025-01-09 20:52:40 on dist-test-slave-612g"
I20250109 20:52:40.193403 564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal/instance:
uuid: "36a29ab2002d4eb389cd11778988e1e9"
format_stamp: "Formatted at 2025-01-09 20:52:40 on dist-test-slave-612g"
I20250109 20:52:40.198510 564 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20250109 20:52:40.201287 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:40.202404 564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250109 20:52:40.202534 564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
uuid: "36a29ab2002d4eb389cd11778988e1e9"
format_stamp: "Formatted at 2025-01-09 20:52:40 on dist-test-slave-612g"
I20250109 20:52:40.202687 564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:52:40.217509 564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:52:40.218228 564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:52:40.218479 564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:52:40.227267 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } attempt: 4
W20250109 20:52:40.229179 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:41583: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111)
I20250109 20:52:40.235879 564 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:41583
I20250109 20:52:40.235898 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:41583 every 8 connection(s)
I20250109 20:52:40.237102 564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
I20250109 20:52:40.240732 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:40.245168 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu as pid 564
I20250109 20:52:40.245324 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal/instance
I20250109 20:52:40.245833 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:52:40.256597 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } attempt: 2
I20250109 20:52:40.258558 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:52:40.266057 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
I20250109 20:52:40.270349 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Bootstrap starting.
I20250109 20:52:40.274250 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:40.275499 490 log.cc:826] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Log is configured to *not* fsync() on all Append() calls
I20250109 20:52:40.277248 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Bootstrap starting.
I20250109 20:52:40.278033 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: No bootstrap required, opened a new log
I20250109 20:52:40.280681 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:40.281605 632 log.cc:826] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Log is configured to *not* fsync() on all Append() calls
I20250109 20:52:40.284081 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: No bootstrap required, opened a new log
I20250109 20:52:40.284642 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.284917 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:40.285018 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8328960890b142248073cec450e957d8, State: Initialized, Role: FOLLOWER
I20250109 20:52:40.285559 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.286337 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.286623 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:52:40.287113 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:52:40.290577 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.290818 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:40.290925 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a29ab2002d4eb389cd11778988e1e9, State: Initialized, Role: FOLLOWER
I20250109 20:52:40.291428 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.292373 646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.292743 646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:52:40.293363 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: configured and running, proceeding with master startup.
W20250109 20:52:40.303485 655 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250109 20:52:40.303592 655 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250109 20:52:40.299993 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:39231: OK, instance_id { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" instance_seqno: 1736455959831780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250109 20:52:40.307036 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:52:40.307590 666 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250109 20:52:40.312157 564 master_runner.cc:186] Error getting master registration for 127.0.104.61:39231: OK, instance_id { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" instance_seqno: 1736455959831780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250109 20:52:40.316426 564 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:52:40.329751 642 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:40.329972 642 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.331209 642 leader_election.cc:290] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231), 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583)
I20250109 20:52:40.331918 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8328960890b142248073cec450e957d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a29ab2002d4eb389cd11778988e1e9" is_pre_election: true
I20250109 20:52:40.332437 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8328960890b142248073cec450e957d8 in term 0.
I20250109 20:52:40.333108 440 leader_election.cc:304] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9, 8328960890b142248073cec450e957d8; no voters:
I20250109 20:52:40.333480 642 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:40.333597 642 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:40.333715 642 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:40.337028 642 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.337756 642 leader_election.cc:290] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [CANDIDATE]: Term 1 election: Requested vote from peers 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231), 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583)
I20250109 20:52:40.338454 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8328960890b142248073cec450e957d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a29ab2002d4eb389cd11778988e1e9"
I20250109 20:52:40.338651 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:40.342226 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8328960890b142248073cec450e957d8 in term 1.
I20250109 20:52:40.342736 440 leader_election.cc:304] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9, 8328960890b142248073cec450e957d8; no voters:
I20250109 20:52:40.343070 642 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:40.344998 642 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8328960890b142248073cec450e957d8, State: Running, Role: LEADER
I20250109 20:52:40.345513 642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.349745 667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8328960890b142248073cec450e957d8. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.350138 667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: LEADER
I20250109 20:52:40.351050 669 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250109 20:52:40.353127 669 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250109 20:52:40.361369 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Refusing update from remote peer 8328960890b142248073cec450e957d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:40.362236 642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:40.368757 646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8328960890b142248073cec450e957d8. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.369751 646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:52:40.374493 646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.374670 646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:52:40.385771 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.386041 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: LEADER
I20250109 20:52:40.389112 669 catalog_manager.cc:1348] Generated new cluster ID: 1852a89fe3204670aafdd57bab642604
I20250109 20:52:40.389252 669 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250109 20:52:40.448887 669 catalog_manager.cc:1371] Generated new certificate authority record
I20250109 20:52:40.450368 669 catalog_manager.cc:1505] Loading token signing keys...
I20250109 20:52:40.466845 669 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Generated new TSK 0
I20250109 20:52:40.468961 669 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250109 20:52:40.483001 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250109 20:52:40.528964 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } attempt: 5
I20250109 20:52:40.540503 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51: Bootstrap starting.
I20250109 20:52:40.544174 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:40.546113 561 log.cc:826] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51: Log is configured to *not* fsync() on all Append() calls
I20250109 20:52:40.549876 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51: No bootstrap required, opened a new log
I20250109 20:52:40.562053 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.562340 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:40.562453 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f066b28f8dd4e9cae5c64bd93feeb51, State: Initialized, Role: FOLLOWER
I20250109 20:52:40.563076 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.564432 678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.565153 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:52:40.565454 534 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:40.568428 534 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Refusing update from remote peer 8328960890b142248073cec450e957d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:40.568884 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8328960890b142248073cec450e957d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:52:40.569293 536 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8328960890b142248073cec450e957d8 in term 1.
I20250109 20:52:40.570092 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8328960890b142248073cec450e957d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51"
W20250109 20:52:40.572017 515 tablet.cc:2367] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:52:40.572736 535 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8328960890b142248073cec450e957d8 in term 1.
I20250109 20:52:40.574064 675 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:40.578495 678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:52:40.603155 684 mvcc.cc:204] Tried to move back new op lower bound from 7112523613630889984 to 7112523613580062720. Current Snapshot: MvccSnapshot[applied={T|T < 7112523613630889984}]
I20250109 20:52:40.612748 678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8328960890b142248073cec450e957d8. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.613072 678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:52:40.612749 681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:40.613216 681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:52:40.631354 694 catalog_manager.cc:1260] Loaded cluster ID: 1852a89fe3204670aafdd57bab642604
I20250109 20:52:40.631453 694 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51: loading cluster ID for follower catalog manager: success
I20250109 20:52:40.634949 694 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51: acquiring CA information for follower catalog manager: success
I20250109 20:52:40.637611 694 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250109 20:52:40.917312 676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:52:40.917656 676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:52:40.917793 676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:52:40.929142 676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:52:40.929432 676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250109 20:52:40.939198 676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:52:40.940649 676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:52:40.942237 676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:52:40.954993 700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:40.955977 701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:40.956790 703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:40.957988 676 server_base.cc:1034] running on GCE node
I20250109 20:52:40.958665 676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:52:40.959839 676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:52:40.961072 676 hybrid_clock.cc:648] HybridClock initialized: now 1736455960961017 us; error 32 us; skew 500 ppm
I20250109 20:52:40.961411 676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:52:40.964061 676 webserver.cc:458] Webserver started at http://127.0.104.1:42763/ using document root <none> and password file <none>
I20250109 20:52:40.964938 676 fs_manager.cc:362] Metadata directory not provided
I20250109 20:52:40.965073 676 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:52:40.965487 676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:52:40.968236 708 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:52:40.968502 708 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:40.969012 676 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/data/instance:
uuid: "09833ea46b7141d58c3899dace69e49f"
format_stamp: "Formatted at 2025-01-09 20:52:40 on dist-test-slave-612g"
I20250109 20:52:40.969918 708 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
I20250109 20:52:40.969925 676 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/instance:
uuid: "09833ea46b7141d58c3899dace69e49f"
format_stamp: "Formatted at 2025-01-09 20:52:40 on dist-test-slave-612g"
I20250109 20:52:40.970453 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:52:40.970822 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:52:40.971541 584 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
I20250109 20:52:40.972146 708 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:52:40.976490 676 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250109 20:52:40.980370 710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:40.981745 676 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250109 20:52:40.981896 676 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal
uuid: "09833ea46b7141d58c3899dace69e49f"
format_stamp: "Formatted at 2025-01-09 20:52:40 on dist-test-slave-612g"
I20250109 20:52:40.982088 676 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:52:41.001276 676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:52:41.002218 676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:52:41.002525 676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:52:41.003626 676 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:52:41.005539 676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:52:41.005625 676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:41.005738 676 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:52:41.005796 676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:41.039631 676 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35465
I20250109 20:52:41.039647 822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35465 every 8 connection(s)
I20250109 20:52:41.041388 676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/data/info.pb
I20250109 20:52:41.043581 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu as pid 676
I20250109 20:52:41.043808 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/instance
I20250109 20:52:41.052593 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250109 20:52:41.086480 681 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:52:41.086685 681 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:41.087543 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:41.087869 823 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:41.088277 681 leader_election.cc:290] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583)
I20250109 20:52:41.089087 823 heartbeater.cc:510] Master 127.0.104.60:41583 requested a full tablet report, sending...
I20250109 20:52:41.089102 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "36a29ab2002d4eb389cd11778988e1e9" is_pre_election: true
I20250109 20:52:41.090219 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:52:41.090720 825 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39231
I20250109 20:52:41.090848 825 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:41.091290 597 ts_manager.cc:194] Registered new tserver with Master: 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:41.091902 825 heartbeater.cc:510] Master 127.0.104.61:39231 requested a full tablet report, sending...
I20250109 20:52:41.092252 512 leader_election.cc:304] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f066b28f8dd4e9cae5c64bd93feeb51; no voters: 36a29ab2002d4eb389cd11778988e1e9, 8328960890b142248073cec450e957d8
I20250109 20:52:41.092650 681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:52:41.094394 526 ts_manager.cc:194] Registered new tserver with Master: 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:41.095738 824 heartbeater.cc:346] Connected to a master server at 127.0.104.62:37941
I20250109 20:52:41.095876 824 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:41.096356 824 heartbeater.cc:510] Master 127.0.104.62:37941 requested a full tablet report, sending...
I20250109 20:52:41.097826 454 ts_manager.cc:194] Registered new tserver with Master: 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:41.099763 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:60669
I20250109 20:52:41.152894 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:41.153281 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: LEADER
I20250109 20:52:41.312319 666 catalog_manager.cc:1260] Loaded cluster ID: 1852a89fe3204670aafdd57bab642604
I20250109 20:52:41.312436 666 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: loading cluster ID for follower catalog manager: success
I20250109 20:52:41.316015 666 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: acquiring CA information for follower catalog manager: success
I20250109 20:52:41.318110 666 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250109 20:52:41.401367 833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:52:41.401628 833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:52:41.401757 833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:52:41.409109 833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:52:41.409327 833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250109 20:52:41.417865 833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:52:41.419286 833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:52:41.420748 833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:52:41.433223 842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:41.434024 843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:41.435292 845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:41.436358 833 server_base.cc:1034] running on GCE node
I20250109 20:52:41.437009 833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:52:41.438027 833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:52:41.439249 833 hybrid_clock.cc:648] HybridClock initialized: now 1736455961439178 us; error 52 us; skew 500 ppm
I20250109 20:52:41.439571 833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:52:41.441998 833 webserver.cc:458] Webserver started at http://127.0.104.2:46385/ using document root <none> and password file <none>
I20250109 20:52:41.442808 833 fs_manager.cc:362] Metadata directory not provided
I20250109 20:52:41.442953 833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:52:41.443293 833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:52:41.446491 833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/data/instance:
uuid: "44d473955919435cb679edd5f8afe2c2"
format_stamp: "Formatted at 2025-01-09 20:52:41 on dist-test-slave-612g"
I20250109 20:52:41.447294 833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/instance:
uuid: "44d473955919435cb679edd5f8afe2c2"
format_stamp: "Formatted at 2025-01-09 20:52:41 on dist-test-slave-612g"
I20250109 20:52:41.453568 833 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20250109 20:52:41.457442 851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:41.458745 833 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250109 20:52:41.458895 833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal
uuid: "44d473955919435cb679edd5f8afe2c2"
format_stamp: "Formatted at 2025-01-09 20:52:41 on dist-test-slave-612g"
I20250109 20:52:41.459071 833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:52:41.496613 833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:52:41.497548 833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:52:41.497840 833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:52:41.498981 833 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:52:41.501528 833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:52:41.501649 833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250109 20:52:41.501754 833 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:52:41.501804 833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:41.533787 833 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42233
I20250109 20:52:41.533789 963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42233 every 8 connection(s)
I20250109 20:52:41.535362 833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/data/info.pb
I20250109 20:52:41.537559 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu as pid 833
I20250109 20:52:41.537777 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/instance
I20250109 20:52:41.542984 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250109 20:52:41.568270 965 heartbeater.cc:346] Connected to a master server at 127.0.104.62:37941
I20250109 20:52:41.568653 965 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:41.569721 965 heartbeater.cc:510] Master 127.0.104.62:37941 requested a full tablet report, sending...
I20250109 20:52:41.570562 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:41.570686 964 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:41.571362 454 ts_manager.cc:194] Registered new tserver with Master: 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:41.571619 964 heartbeater.cc:510] Master 127.0.104.60:41583 requested a full tablet report, sending...
I20250109 20:52:41.572360 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:33073
I20250109 20:52:41.572789 597 ts_manager.cc:194] Registered new tserver with Master: 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:41.574765 966 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39231
I20250109 20:52:41.574889 966 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:41.575199 966 heartbeater.cc:510] Master 127.0.104.61:39231 requested a full tablet report, sending...
I20250109 20:52:41.576299 526 ts_manager.cc:194] Registered new tserver with Master: 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:41.602905 824 heartbeater.cc:502] Master 127.0.104.62:37941 was elected leader, sending a full tablet report...
I20250109 20:52:41.655248 972 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:52:41.655454 972 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:41.656679 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "36a29ab2002d4eb389cd11778988e1e9" is_pre_election: true
I20250109 20:52:41.656690 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:52:41.657434 512 leader_election.cc:304] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f066b28f8dd4e9cae5c64bd93feeb51; no voters: 36a29ab2002d4eb389cd11778988e1e9, 8328960890b142248073cec450e957d8
I20250109 20:52:41.657559 972 leader_election.cc:290] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583)
I20250109 20:52:41.657722 972 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 0f066b28f8dd4e9cae5c64bd93feeb51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:52:41.886955 970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:52:41.887249 970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:52:41.887342 970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:52:41.894536 970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:52:41.894747 970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250109 20:52:41.903348 970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:52:41.905303 970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:52:41.907127 970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250109 20:52:41.919795 978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:41.920496 979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:41.921761 970 server_base.cc:1034] running on GCE node
W20250109 20:52:41.922580 981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:41.923177 970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:52:41.924033 970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:52:41.925218 970 hybrid_clock.cc:648] HybridClock initialized: now 1736455961925165 us; error 39 us; skew 500 ppm
I20250109 20:52:41.925496 970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:52:41.927981 970 webserver.cc:458] Webserver started at http://127.0.104.3:44091/ using document root <none> and password file <none>
I20250109 20:52:41.928670 970 fs_manager.cc:362] Metadata directory not provided
I20250109 20:52:41.928836 970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:52:41.929126 970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250109 20:52:41.931444 970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/data/instance:
uuid: "5fcc647ed479491b88e5ffacf2927a1a"
format_stamp: "Formatted at 2025-01-09 20:52:41 on dist-test-slave-612g"
I20250109 20:52:41.932083 970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/instance:
uuid: "5fcc647ed479491b88e5ffacf2927a1a"
format_stamp: "Formatted at 2025-01-09 20:52:41 on dist-test-slave-612g"
I20250109 20:52:41.936808 970 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20250109 20:52:41.939728 987 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:41.940896 970 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250109 20:52:41.941035 970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal
uuid: "5fcc647ed479491b88e5ffacf2927a1a"
format_stamp: "Formatted at 2025-01-09 20:52:41 on dist-test-slave-612g"
I20250109 20:52:41.941193 970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:52:41.952385 970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:52:41.953181 970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:52:41.953433 970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:52:41.954283 970 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250109 20:52:41.955559 970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250109 20:52:41.955633 970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:41.955753 970 ts_tablet_manager.cc:610] Registered 0 tablets
I20250109 20:52:41.955801 970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:41.986403 1096 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:52:41.986557 1096 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:41.987592 1096 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
I20250109 20:52:41.987732 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:52:41.987860 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:52:41.988490 584 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
I20250109 20:52:41.988862 1096 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:52:41.989405 970 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36949
I20250109 20:52:41.989444 1100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36949 every 8 connection(s)
I20250109 20:52:41.990530 970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/data/info.pb
I20250109 20:52:41.994581 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu as pid 970
I20250109 20:52:41.994755 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/instance
I20250109 20:52:42.010614 1102 heartbeater.cc:346] Connected to a master server at 127.0.104.62:37941
I20250109 20:52:42.010970 1102 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:42.011953 1102 heartbeater.cc:510] Master 127.0.104.62:37941 requested a full tablet report, sending...
I20250109 20:52:42.013532 454 ts_manager.cc:194] Registered new tserver with Master: 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:42.014151 1103 heartbeater.cc:346] Connected to a master server at 127.0.104.61:39231
I20250109 20:52:42.014285 1103 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:42.014487 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:46485
I20250109 20:52:42.014667 1103 heartbeater.cc:510] Master 127.0.104.61:39231 requested a full tablet report, sending...
I20250109 20:52:42.015584 526 ts_manager.cc:194] Registered new tserver with Master: 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:42.016484 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:42.016608 1101 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:42.016961 1101 heartbeater.cc:510] Master 127.0.104.60:41583 requested a full tablet report, sending...
I20250109 20:52:42.017925 597 ts_manager.cc:194] Registered new tserver with Master: 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:42.020435 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250109 20:52:42.045660 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_37eedb8517aa4f89a6cfc5b1b3f07b1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.045698 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2e5f440a1e9a4e6083d0857464311f92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.046991 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37eedb8517aa4f89a6cfc5b1b3f07b1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:42.047046 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e5f440a1e9a4e6083d0857464311f92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.049466 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e43084f782e542dca1794001496f1f49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.050058 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e43084f782e542dca1794001496f1f49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.050788 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c0c3b778a0fc41ec99a9a38a35765cd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.051313 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0c3b778a0fc41ec99a9a38a35765cd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.073446 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_81bc625ee9694d2ba87e1dafcbcad1f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.074226 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81bc625ee9694d2ba87e1dafcbcad1f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.075064 965 heartbeater.cc:502] Master 127.0.104.62:37941 was elected leader, sending a full tablet report...
I20250109 20:52:42.075320 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7204ac0d60734e9ba819bfac04c4b221"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.075831 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7204ac0d60734e9ba819bfac04c4b221 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.076412 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1b67d2a3c0b3403b8041cfb32c37b227"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.077071 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b67d2a3c0b3403b8041cfb32c37b227 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.077702 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9849b6bfd6ab46f7ae94a43bafe5ec11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.081377 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9849b6bfd6ab46f7ae94a43bafe5ec11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.086473 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_eb403509cf414f548af84d837a6f04ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.086913 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb403509cf414f548af84d837a6f04ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.088423 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_37eedb8517aa4f89a6cfc5b1b3f07b1c" } new_table_name: "default.Table_19ca97c8cc794f37879540b3843a65a0" modify_external_catalogs: true
I20250109 20:52:42.084337 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3c6926f2bfda432ca54821bb09fb463a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.091182 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c6926f2bfda432ca54821bb09fb463a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:42.139405 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 103722f5a2314bbeb9b141bb0a460ece consensus state has no leader
I20250109 20:52:42.142786 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e43084f782e542dca1794001496f1f49" } new_table_name: "default.Table_732fb1f4c69449e8950e5fa85c7a2948" modify_external_catalogs: true
I20250109 20:52:42.144389 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_19ca97c8cc794f37879540b3843a65a0" } new_table_name: "default.Table_0710fdbfdd7a4e7b892b501e77fc2e72" modify_external_catalogs: true
I20250109 20:52:42.145885 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7204ac0d60734e9ba819bfac04c4b221" } new_table_name: "default.Table_1f430949a522426db107b249ce1ab5f0" modify_external_catalogs: true
I20250109 20:52:42.147265 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c0c3b778a0fc41ec99a9a38a35765cd4" } new_table_name: "default.Table_70760925d6df4f0c961c825337270e1a" modify_external_catalogs: true
W20250109 20:52:42.163327 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 899f08250f2248778f6abab551093461 has no consensus state
W20250109 20:52:42.163753 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 103722f5a2314bbeb9b141bb0a460ece consensus state has no leader
W20250109 20:52:42.165510 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e784a5378f7740baa39673071b56d8c5 has no consensus state
W20250109 20:52:42.165973 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c38c8f0299e49ff8aff99b17b7be01b has no consensus state
W20250109 20:52:42.169663 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 103722f5a2314bbeb9b141bb0a460ece (table default.Table_0710fdbfdd7a4e7b892b501e77fc2e72 [id=606e0f04684243e8a30813edb6e96766], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 103722f5a2314bbeb9b141bb0a460ece consensus state has no leader
I20250109 20:52:42.185947 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0710fdbfdd7a4e7b892b501e77fc2e72" } modify_external_catalogs: true
I20250109 20:52:42.186211 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0710fdbfdd7a4e7b892b501e77fc2e72" } modify_external_catalogs: true
I20250109 20:52:42.187683 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_70760925d6df4f0c961c825337270e1a" } modify_external_catalogs: true
I20250109 20:52:42.187981 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_70760925d6df4f0c961c825337270e1a" } modify_external_catalogs: true
I20250109 20:52:42.190856 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e5f440a1e9a4e6083d0857464311f92" } modify_external_catalogs: true
I20250109 20:52:42.191020 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e5f440a1e9a4e6083d0857464311f92" } modify_external_catalogs: true
I20250109 20:52:42.198143 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ba365ba04f6848328a9b14dc4c92ac2d
I20250109 20:52:42.202373 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6c38c8f0299e49ff8aff99b17b7be01b
I20250109 20:52:42.199779 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.201109 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b (DEFAULT_TABLE table=default.Table_70760925d6df4f0c961c825337270e1a [id=d9fb05f1ceef49638bbe6db78d2d17c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.204421 750 tablet_service.cc:1467] Processing CreateTablet for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (DEFAULT_TABLE table=default.Table_81bc625ee9694d2ba87e1dafcbcad1f9 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.205252 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c38c8f0299e49ff8aff99b17b7be01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.205782 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.202342 756 tablet_service.cc:1467] Processing CreateTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 (DEFAULT_TABLE table=default.Table_eb403509cf414f548af84d837a6f04ac [id=a1dea7259a764d7ca62d5ff76ccc95b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.207430 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920a6be9db344c098c62bfd2fa5a73e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.207975 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 103722f5a2314bbeb9b141bb0a460ece
I20250109 20:52:42.201428 754 tablet_service.cc:1467] Processing CreateTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d (DEFAULT_TABLE table=default.Table_2e5f440a1e9a4e6083d0857464311f92 [id=613376bfcfe64a468bcc9dc242929efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.208838 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba365ba04f6848328a9b14dc4c92ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.202121 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8389e66e581f406bb82735a1ece78773 (DEFAULT_TABLE table=default.Table_9849b6bfd6ab46f7ae94a43bafe5ec11 [id=d641ab17c06741d4a0816205d1f4b16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.213436 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8389e66e581f406bb82735a1ece78773. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.202541 757 tablet_service.cc:1467] Processing CreateTablet for tablet 30ae6c4e835048bd8deed5ac9c4ae07a (DEFAULT_TABLE table=default.Table_3c6926f2bfda432ca54821bb09fb463a [id=6fe0e6310b9f44a8b3b9558fec22c2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.214965 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ae6c4e835048bd8deed5ac9c4ae07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.202721 758 tablet_service.cc:1467] Processing CreateTablet for tablet 103722f5a2314bbeb9b141bb0a460ece (DEFAULT_TABLE table=default.Table_37eedb8517aa4f89a6cfc5b1b3f07b1c [id=606e0f04684243e8a30813edb6e96766]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.217170 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103722f5a2314bbeb9b141bb0a460ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.220335 755 tablet_service.cc:1467] Processing CreateTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 (DEFAULT_TABLE table=default.Table_1b67d2a3c0b3403b8041cfb32c37b227 [id=e68ca2b1501a493a97b5ae61bdd9fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.220845 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be88a4cf79c14c0fb3ab1110a3fa1714. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.227224 1158 tablet_bootstrap.cc:492] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:42.241192 753 tablet_service.cc:1467] Processing CreateTablet for tablet e784a5378f7740baa39673071b56d8c5 (DEFAULT_TABLE table=default.Table_1f430949a522426db107b249ce1ab5f0 [id=e4a51641b240493d9f69b9031ef5d4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.241636 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e784a5378f7740baa39673071b56d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.243854 754 tablet_service.cc:1467] Processing CreateTablet for tablet 899f08250f2248778f6abab551093461 (DEFAULT_TABLE table=default.Table_732fb1f4c69449e8950e5fa85c7a2948 [id=a667d9b2080d44b89a1715b9b57c8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.244388 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899f08250f2248778f6abab551093461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.248947 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 103722f5a2314bbeb9b141bb0a460ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.249275 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:42.250550 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ba365ba04f6848328a9b14dc4c92ac2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba365ba04f6848328a9b14dc4c92ac2d already in progress: creating tablet
W20250109 20:52:42.250932 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 103722f5a2314bbeb9b141bb0a460ece because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 103722f5a2314bbeb9b141bb0a460ece already in progress: creating tablet
W20250109 20:52:42.251191 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6c38c8f0299e49ff8aff99b17b7be01b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c38c8f0299e49ff8aff99b17b7be01b already in progress: creating tablet
I20250109 20:52:42.253057 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_eb403509cf414f548af84d837a6f04ac" } modify_external_catalogs: true
I20250109 20:52:42.253296 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_eb403509cf414f548af84d837a6f04ac" } modify_external_catalogs: true
I20250109 20:52:42.259274 1158 tablet_bootstrap.cc:654] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.260641 1158 log.cc:826] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: Log is configured to *not* fsync() on all Append() calls
I20250109 20:52:42.261255 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.261806 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 920a6be9db344c098c62bfd2fa5a73e8
I20250109 20:52:42.262290 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 103722f5a2314bbeb9b141bb0a460ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.263203 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:42.264194 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 920a6be9db344c098c62bfd2fa5a73e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 920a6be9db344c098c62bfd2fa5a73e8 already in progress: creating tablet
I20250109 20:52:42.267155 895 tablet_service.cc:1467] Processing CreateTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d (DEFAULT_TABLE table=default.Table_2e5f440a1e9a4e6083d0857464311f92 [id=613376bfcfe64a468bcc9dc242929efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.272217 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_732fb1f4c69449e8950e5fa85c7a2948" } modify_external_catalogs: true
I20250109 20:52:42.272450 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_732fb1f4c69449e8950e5fa85c7a2948" } modify_external_catalogs: true
I20250109 20:52:42.272548 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1d87425867fc462e86d1f51d3c7777dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.259230 893 tablet_service.cc:1514] Processing DeleteTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.268712 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba365ba04f6848328a9b14dc4c92ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:42.273147 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d87425867fc462e86d1f51d3c7777dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.268965 896 tablet_service.cc:1467] Processing CreateTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 (DEFAULT_TABLE table=default.Table_1b67d2a3c0b3403b8041cfb32c37b227 [id=e68ca2b1501a493a97b5ae61bdd9fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.277341 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be88a4cf79c14c0fb3ab1110a3fa1714. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.269078 891 tablet_service.cc:1467] Processing CreateTablet for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (DEFAULT_TABLE table=default.Table_81bc625ee9694d2ba87e1dafcbcad1f9 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.278558 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.269174 888 tablet_service.cc:1467] Processing CreateTablet for tablet e784a5378f7740baa39673071b56d8c5 (DEFAULT_TABLE table=default.Table_1f430949a522426db107b249ce1ab5f0 [id=e4a51641b240493d9f69b9031ef5d4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.279621 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e784a5378f7740baa39673071b56d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.269266 894 tablet_service.cc:1467] Processing CreateTablet for tablet 8389e66e581f406bb82735a1ece78773 (DEFAULT_TABLE table=default.Table_9849b6bfd6ab46f7ae94a43bafe5ec11 [id=d641ab17c06741d4a0816205d1f4b16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.280618 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8389e66e581f406bb82735a1ece78773. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.269359 889 tablet_service.cc:1467] Processing CreateTablet for tablet 899f08250f2248778f6abab551093461 (DEFAULT_TABLE table=default.Table_732fb1f4c69449e8950e5fa85c7a2948 [id=a667d9b2080d44b89a1715b9b57c8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.281899 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899f08250f2248778f6abab551093461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.282131 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 899f08250f2248778f6abab551093461
I20250109 20:52:42.269451 892 tablet_service.cc:1467] Processing CreateTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b (DEFAULT_TABLE table=default.Table_70760925d6df4f0c961c825337270e1a [id=d9fb05f1ceef49638bbe6db78d2d17c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.282945 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c38c8f0299e49ff8aff99b17b7be01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.269544 897 tablet_service.cc:1467] Processing CreateTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 (DEFAULT_TABLE table=default.Table_eb403509cf414f548af84d837a6f04ac [id=a1dea7259a764d7ca62d5ff76ccc95b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.284114 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920a6be9db344c098c62bfd2fa5a73e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.285530 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 899f08250f2248778f6abab551093461 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.269639 899 tablet_service.cc:1467] Processing CreateTablet for tablet 103722f5a2314bbeb9b141bb0a460ece (DEFAULT_TABLE table=default.Table_37eedb8517aa4f89a6cfc5b1b3f07b1c [id=606e0f04684243e8a30813edb6e96766]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.287303 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103722f5a2314bbeb9b141bb0a460ece. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:42.287981 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 899f08250f2248778f6abab551093461 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 899f08250f2248778f6abab551093461 already in progress: creating tablet
I20250109 20:52:42.288215 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bf5f916b37774a7e901eadf1fe08c913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.288599 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 899f08250f2248778f6abab551093461 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.288821 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:42.288908 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf5f916b37774a7e901eadf1fe08c913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.266559 898 tablet_service.cc:1467] Processing CreateTablet for tablet 30ae6c4e835048bd8deed5ac9c4ae07a (DEFAULT_TABLE table=default.Table_3c6926f2bfda432ca54821bb09fb463a [id=6fe0e6310b9f44a8b3b9558fec22c2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.289619 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ae6c4e835048bd8deed5ac9c4ae07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.303637 753 tablet_service.cc:1467] Processing CreateTablet for tablet 75ec6b6b1ec1462d82008a78a885032d (DEFAULT_TABLE table=default.Table_1d87425867fc462e86d1f51d3c7777dd [id=289e52afdcea437bb60cd657c8dc01d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.304123 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ec6b6b1ec1462d82008a78a885032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.304467 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bece8c2a3b4c4f42981b775b7b6ec654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.305006 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bece8c2a3b4c4f42981b775b7b6ec654 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.307801 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_697b2fe8a4554d3585e94d97c2cfe0e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.308288 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_697b2fe8a4554d3585e94d97c2cfe0e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.315877 754 tablet_service.cc:1467] Processing CreateTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 (DEFAULT_TABLE table=default.Table_bf5f916b37774a7e901eadf1fe08c913 [id=315670c6ced1467e82fc224f50ca5625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.316273 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecee7292068c4d649a1232ac54dbb7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.321631 893 tablet_service.cc:1467] Processing CreateTablet for tablet 75ec6b6b1ec1462d82008a78a885032d (DEFAULT_TABLE table=default.Table_1d87425867fc462e86d1f51d3c7777dd [id=289e52afdcea437bb60cd657c8dc01d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.322121 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ec6b6b1ec1462d82008a78a885032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.323706 887 tablet_service.cc:1467] Processing CreateTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 (DEFAULT_TABLE table=default.Table_bf5f916b37774a7e901eadf1fe08c913 [id=315670c6ced1467e82fc224f50ca5625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.324374 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecee7292068c4d649a1232ac54dbb7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.333003 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_96c90b1435444234b7c6d3104723f4be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.333693 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96c90b1435444234b7c6d3104723f4be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.336752 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8c0e3ac68686429dadf4fcc158cf285b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.337224 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c0e3ac68686429dadf4fcc158cf285b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.340250 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_38a61ab14f8c46f0b9bd21c3d95c9ac4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.340888 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38a61ab14f8c46f0b9bd21c3d95c9ac4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.349273 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_df0d03a66d1643bca34040c9aa2e1f3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.349869 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df0d03a66d1643bca34040c9aa2e1f3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.351279 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_401fcdac6f4d4150b7acedb7e3c8790e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.351816 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_401fcdac6f4d4150b7acedb7e3c8790e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.355995 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3c6926f2bfda432ca54821bb09fb463a" } new_table_name: "default.Table_a6d4d40a237548c384fd897eb207903c" modify_external_catalogs: true
I20250109 20:52:42.357975 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dab2393d93b14ec79d96807a37360e33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.358443 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dab2393d93b14ec79d96807a37360e33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.361146 753 tablet_service.cc:1467] Processing CreateTablet for tablet 0454b27918ab41ad9ab269185b2692fe (DEFAULT_TABLE table=default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.361617 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0454b27918ab41ad9ab269185b2692fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.366577 886 tablet_service.cc:1467] Processing CreateTablet for tablet 0454b27918ab41ad9ab269185b2692fe (DEFAULT_TABLE table=default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.367149 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0454b27918ab41ad9ab269185b2692fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.371163 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bf5f916b37774a7e901eadf1fe08c913" } new_table_name: "default.Table_7a8859a022a54a48affdc0ef3024c194" modify_external_catalogs: true
I20250109 20:52:42.371261 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9849b6bfd6ab46f7ae94a43bafe5ec11" } new_table_name: "default.Table_39dd6100bb4044f6a89213a6875f8756" modify_external_catalogs: true
I20250109 20:52:42.387449 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_697b2fe8a4554d3585e94d97c2cfe0e7" } new_table_name: "default.Table_e06f920c44304e0c806a3aa165fde5d0" modify_external_catalogs: true
I20250109 20:52:42.389348 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_96c90b1435444234b7c6d3104723f4be" } new_table_name: "default.Table_3eef1bf244124c8d8d9bda7d0699a6f7" modify_external_catalogs: true
W20250109 20:52:42.398567 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ecee7292068c4d649a1232ac54dbb7f0 consensus state has no leader
W20250109 20:52:42.398864 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30ae6c4e835048bd8deed5ac9c4ae07a consensus state has no leader
W20250109 20:52:42.400071 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8389e66e581f406bb82735a1ece78773 consensus state has no leader
I20250109 20:52:42.401378 753 tablet_service.cc:1467] Processing CreateTablet for tablet cdd067c335354357b44dac433f8c1523 (DEFAULT_TABLE table=default.Table_697b2fe8a4554d3585e94d97c2cfe0e7 [id=8c860f8c96254692906f892a25530962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.401785 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd067c335354357b44dac433f8c1523. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.401928 885 tablet_service.cc:1467] Processing CreateTablet for tablet cdd067c335354357b44dac433f8c1523 (DEFAULT_TABLE table=default.Table_697b2fe8a4554d3585e94d97c2cfe0e7 [id=8c860f8c96254692906f892a25530962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.402388 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd067c335354357b44dac433f8c1523. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.419489 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1b67d2a3c0b3403b8041cfb32c37b227" } modify_external_catalogs: true
I20250109 20:52:42.419688 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1b67d2a3c0b3403b8041cfb32c37b227" } modify_external_catalogs: true
I20250109 20:52:42.420334 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d87425867fc462e86d1f51d3c7777dd" } modify_external_catalogs: true
I20250109 20:52:42.420456 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d87425867fc462e86d1f51d3c7777dd" } modify_external_catalogs: true
I20250109 20:52:42.423395 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 103722f5a2314bbeb9b141bb0a460ece (DEFAULT_TABLE table=default.Table_37eedb8517aa4f89a6cfc5b1b3f07b1c [id=606e0f04684243e8a30813edb6e96766]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.426195 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 30ae6c4e835048bd8deed5ac9c4ae07a (DEFAULT_TABLE table=default.Table_3c6926f2bfda432ca54821bb09fb463a [id=6fe0e6310b9f44a8b3b9558fec22c2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.426231 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 (DEFAULT_TABLE table=default.Table_eb403509cf414f548af84d837a6f04ac [id=a1dea7259a764d7ca62d5ff76ccc95b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.426592 1165 tablet_bootstrap.cc:492] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:42.426779 1031 tablet_service.cc:1467] Processing CreateTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d (DEFAULT_TABLE table=default.Table_2e5f440a1e9a4e6083d0857464311f92 [id=613376bfcfe64a468bcc9dc242929efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.426898 1032 tablet_service.cc:1467] Processing CreateTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 (DEFAULT_TABLE table=default.Table_1b67d2a3c0b3403b8041cfb32c37b227 [id=e68ca2b1501a493a97b5ae61bdd9fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.427433 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 8389e66e581f406bb82735a1ece78773 (DEFAULT_TABLE table=default.Table_9849b6bfd6ab46f7ae94a43bafe5ec11 [id=d641ab17c06741d4a0816205d1f4b16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.428001 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b (DEFAULT_TABLE table=default.Table_70760925d6df4f0c961c825337270e1a [id=d9fb05f1ceef49638bbe6db78d2d17c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.428474 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (DEFAULT_TABLE table=default.Table_81bc625ee9694d2ba87e1dafcbcad1f9 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.428932 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.429054 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 899f08250f2248778f6abab551093461 (DEFAULT_TABLE table=default.Table_732fb1f4c69449e8950e5fa85c7a2948 [id=a667d9b2080d44b89a1715b9b57c8e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.431275 1024 tablet_service.cc:1467] Processing CreateTablet for tablet e784a5378f7740baa39673071b56d8c5 (DEFAULT_TABLE table=default.Table_1f430949a522426db107b249ce1ab5f0 [id=e4a51641b240493d9f69b9031ef5d4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.431536 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 103722f5a2314bbeb9b141bb0a460ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.431819 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.432133 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 899f08250f2248778f6abab551093461 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.432289 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 75ec6b6b1ec1462d82008a78a885032d (DEFAULT_TABLE table=default.Table_1d87425867fc462e86d1f51d3c7777dd [id=289e52afdcea437bb60cd657c8dc01d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.432615 1019 tablet_service.cc:1467] Processing CreateTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 (DEFAULT_TABLE table=default.Table_bf5f916b37774a7e901eadf1fe08c913 [id=315670c6ced1467e82fc224f50ca5625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.433091 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 0454b27918ab41ad9ab269185b2692fe (DEFAULT_TABLE table=default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.433954 1017 tablet_service.cc:1467] Processing CreateTablet for tablet cdd067c335354357b44dac433f8c1523 (DEFAULT_TABLE table=default.Table_697b2fe8a4554d3585e94d97c2cfe0e7 [id=8c860f8c96254692906f892a25530962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.427227 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.434859 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103722f5a2314bbeb9b141bb0a460ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.435380 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8389e66e581f406bb82735a1ece78773. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.436908 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899f08250f2248778f6abab551093461. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:42.437796 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdd067c335354357b44dac433f8c1523 consensus state has no leader
I20250109 20:52:42.437959 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e784a5378f7740baa39673071b56d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.438189 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be88a4cf79c14c0fb3ab1110a3fa1714. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.439561 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecee7292068c4d649a1232ac54dbb7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.445295 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd067c335354357b44dac433f8c1523. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.446833 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.448850 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba365ba04f6848328a9b14dc4c92ac2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:42.453012 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26f5b4af510942c8b50838df3bc8d8fc has no consensus state
I20250109 20:52:42.456997 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7a8859a022a54a48affdc0ef3024c194" } modify_external_catalogs: true
I20250109 20:52:42.457219 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7a8859a022a54a48affdc0ef3024c194" } modify_external_catalogs: true
I20250109 20:52:42.458933 1158 tablet_bootstrap.cc:492] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:42.459249 1158 ts_tablet_manager.cc:1397] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.232s user 0.067s sys 0.063s
I20250109 20:52:42.459762 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c38c8f0299e49ff8aff99b17b7be01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.460722 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920a6be9db344c098c62bfd2fa5a73e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.466980 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dab2393d93b14ec79d96807a37360e33" } modify_external_catalogs: true
I20250109 20:52:42.467181 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dab2393d93b14ec79d96807a37360e33" } modify_external_catalogs: true
I20250109 20:52:42.479295 754 tablet_service.cc:1467] Processing CreateTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 (DEFAULT_TABLE table=default.Table_dab2393d93b14ec79d96807a37360e33 [id=fcec4978e3584a9cabd5641b5ffb7340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.479809 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a1a6a0a1f4411aa0b57481b6291808. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.489503 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_39dd6100bb4044f6a89213a6875f8756" } modify_external_catalogs: true
I20250109 20:52:42.489758 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_39dd6100bb4044f6a89213a6875f8756" } modify_external_catalogs: true
I20250109 20:52:42.489646 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7cbb1b8e12d44f4198d361e804254e84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.490422 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cbb1b8e12d44f4198d361e804254e84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.487500 1158 raft_consensus.cc:357] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.495404 1158 raft_consensus.cc:383] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.495538 1158 raft_consensus.cc:738] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.499078 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_709427497f3a436a99bee6129d87efb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.499792 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_709427497f3a436a99bee6129d87efb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.500568 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu pid 564
I20250109 20:52:42.501792 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20250109 20:52:42.502259 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ec6b6b1ec1462d82008a78a885032d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.498036 1158 consensus_queue.cc:260] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.512248 1158 ts_tablet_manager.cc:1428] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.053s user 0.016s sys 0.004s
I20250109 20:52:42.514570 1158 tablet_bootstrap.cc:492] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:42.516356 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ae6c4e835048bd8deed5ac9c4ae07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.516571 1158 tablet_bootstrap.cc:654] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.517499 1102 heartbeater.cc:502] Master 127.0.104.62:37941 was elected leader, sending a full tablet report...
I20250109 20:52:42.517019 1178 raft_consensus.cc:491] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:42.518949 1178 raft_consensus.cc:513] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:42.525625 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6c38c8f0299e49ff8aff99b17b7be01b because the tablet was not found. No further retry: Not found: Tablet not found: 6c38c8f0299e49ff8aff99b17b7be01b
W20250109 20:52:42.526135 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 103722f5a2314bbeb9b141bb0a460ece because the tablet was not found. No further retry: Not found: Tablet not found: 103722f5a2314bbeb9b141bb0a460ece
W20250109 20:52:42.526369 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 920a6be9db344c098c62bfd2fa5a73e8 because the tablet was not found. No further retry: Not found: Tablet not found: 920a6be9db344c098c62bfd2fa5a73e8
W20250109 20:52:42.527110 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 899f08250f2248778f6abab551093461 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 899f08250f2248778f6abab551093461 already in progress: creating tablet
W20250109 20:52:42.527334 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ba365ba04f6848328a9b14dc4c92ac2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba365ba04f6848328a9b14dc4c92ac2d already in progress: creating tablet
I20250109 20:52:42.529536 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0454b27918ab41ad9ab269185b2692fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.533466 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_49a9931b8903487db0856c10d60fc1e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.534072 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49a9931b8903487db0856c10d60fc1e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.539117 1176 tablet_bootstrap.cc:492] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:42.543349 1176 tablet_bootstrap.cc:654] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.546259 1158 tablet_bootstrap.cc:492] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:42.546407 1158 ts_tablet_manager.cc:1397] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20250109 20:52:42.547360 1158 raft_consensus.cc:357] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.547585 1158 raft_consensus.cc:383] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.547664 1158 raft_consensus.cc:738] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.547919 1158 consensus_queue.cc:260] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.549080 1158 ts_tablet_manager.cc:1428] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:42.549330 1158 tablet_bootstrap.cc:492] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:42.550987 1158 tablet_bootstrap.cc:654] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.561371 1178 leader_election.cc:290] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:42.562999 1025 tablet_service.cc:1467] Processing CreateTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 (DEFAULT_TABLE table=default.Table_dab2393d93b14ec79d96807a37360e33 [id=fcec4978e3584a9cabd5641b5ffb7340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.563349 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a1a6a0a1f4411aa0b57481b6291808. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.564410 753 tablet_service.cc:1467] Processing CreateTablet for tablet 6bea28fb446e49669133e21b02c416ad (DEFAULT_TABLE table=default.Table_401fcdac6f4d4150b7acedb7e3c8790e [id=06cecc207bb746dd9b0cb0f6eac4017e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.564823 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bea28fb446e49669133e21b02c416ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.567329 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 58817dddbe8749249a96c60dee41b520 (DEFAULT_TABLE table=default.Table_38a61ab14f8c46f0b9bd21c3d95c9ac4 [id=fc46d1a61d7d4769bfcf249712d2e2b9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:42.567466 830 tablet.cc:2367] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:52:42.567636 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58817dddbe8749249a96c60dee41b520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.568388 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 6bea28fb446e49669133e21b02c416ad (DEFAULT_TABLE table=default.Table_401fcdac6f4d4150b7acedb7e3c8790e [id=06cecc207bb746dd9b0cb0f6eac4017e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.568749 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bea28fb446e49669133e21b02c416ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.570560 1027 tablet_service.cc:1467] Processing CreateTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d (DEFAULT_TABLE table=default.Table_df0d03a66d1643bca34040c9aa2e1f3e [id=ae7fb9326c2c49a593d1111d45ea424f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.570897 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d6dc79bf1e48ee9cf2bbb70463741d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.572921 752 tablet_service.cc:1467] Processing CreateTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d (DEFAULT_TABLE table=default.Table_df0d03a66d1643bca34040c9aa2e1f3e [id=ae7fb9326c2c49a593d1111d45ea424f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.573118 753 tablet_service.cc:1467] Processing CreateTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d (DEFAULT_TABLE table=default.Table_8c0e3ac68686429dadf4fcc158cf285b [id=0acf665b047a4f1dac232ef98359b372]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.573287 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d6dc79bf1e48ee9cf2bbb70463741d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.573458 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2aacfc7eea3490dbeb9d5c82d48766d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.573804 757 tablet_service.cc:1467] Processing CreateTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc (DEFAULT_TABLE table=default.Table_3eef1bf244124c8d8d9bda7d0699a6f7 [id=0048f4157ceb4c098d924be9004eeaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.574167 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f5b4af510942c8b50838df3bc8d8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.572921 751 tablet_service.cc:1467] Processing CreateTablet for tablet 58817dddbe8749249a96c60dee41b520 (DEFAULT_TABLE table=default.Table_38a61ab14f8c46f0b9bd21c3d95c9ac4 [id=fc46d1a61d7d4769bfcf249712d2e2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.575729 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58817dddbe8749249a96c60dee41b520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.577008 1028 tablet_service.cc:1467] Processing CreateTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d (DEFAULT_TABLE table=default.Table_8c0e3ac68686429dadf4fcc158cf285b [id=0acf665b047a4f1dac232ef98359b372]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.577291 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2aacfc7eea3490dbeb9d5c82d48766d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.577668 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 75ec6b6b1ec1462d82008a78a885032d
I20250109 20:52:42.578467 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e1e2df6604b3481583f4036b3aee752e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.578922 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1e2df6604b3481583f4036b3aee752e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.580821 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ec6b6b1ec1462d82008a78a885032d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:42.581677 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 75ec6b6b1ec1462d82008a78a885032d because the tablet was not found. No further retry: Not found: Tablet not found: 75ec6b6b1ec1462d82008a78a885032d
I20250109 20:52:42.584448 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ecee7292068c4d649a1232ac54dbb7f0
I20250109 20:52:42.585065 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5c35bca9b68e42bb9075f51687f74d15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.585474 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c35bca9b68e42bb9075f51687f74d15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.586599 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f6a1a6a0a1f4411aa0b57481b6291808
I20250109 20:52:42.587128 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_60febb828b3c4979bf94fd0f01bdfe4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.587519 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60febb828b3c4979bf94fd0f01bdfe4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.588889 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet be88a4cf79c14c0fb3ab1110a3fa1714
I20250109 20:52:42.589612 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_130e9f2c3d5c42b8b1147cbf8a482014"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.591104 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_1f430949a522426db107b249ce1ab5f0 [id=e4a51641b240493d9f69b9031ef5d4b1]). Expected version 1 got 0
I20250109 20:52:42.591465 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.592011 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8389e66e581f406bb82735a1ece78773
I20250109 20:52:42.593575 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9af6ca0c3f0042358b13f0891ced6dea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.594102 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9af6ca0c3f0042358b13f0891ced6dea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.595528 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9e3ee29227aa4e07b4f331683046e066"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.596045 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e3ee29227aa4e07b4f331683046e066 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.596501 1176 log.cc:826] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: Log is configured to *not* fsync() on all Append() calls
I20250109 20:52:42.599161 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.599586 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc (DEFAULT_TABLE table=default.Table_3eef1bf244124c8d8d9bda7d0699a6f7 [id=0048f4157ceb4c098d924be9004eeaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.599965 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f5b4af510942c8b50838df3bc8d8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.600988 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 899f08250f2248778f6abab551093461 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.604429 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ec6b6b1ec1462d82008a78a885032d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.604956 756 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:42.590056 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_130e9f2c3d5c42b8b1147cbf8a482014 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.605487 755 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.606711 750 tablet_service.cc:1514] Processing DeleteTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:42.610756 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 75ec6b6b1ec1462d82008a78a885032d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ec6b6b1ec1462d82008a78a885032d already in progress: creating tablet
I20250109 20:52:42.610805 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_02728ead2eff4bd4b7809e4da74fe753"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.611229 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ecee7292068c4d649a1232ac54dbb7f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecee7292068c4d649a1232ac54dbb7f0 already in progress: creating tablet
W20250109 20:52:42.611294 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02728ead2eff4bd4b7809e4da74fe753 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:42.611584 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f6a1a6a0a1f4411aa0b57481b6291808 because the tablet was not found. No further retry: Not found: Tablet not found: f6a1a6a0a1f4411aa0b57481b6291808
W20250109 20:52:42.611940 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet be88a4cf79c14c0fb3ab1110a3fa1714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be88a4cf79c14c0fb3ab1110a3fa1714 already in progress: creating tablet
I20250109 20:52:42.612386 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8389e66e581f406bb82735a1ece78773 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.616251 1157 tablet_replica.cc:331] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:42.616788 1157 raft_consensus.cc:2238] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.617720 1157 raft_consensus.cc:2267] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.621687 891 tablet_service.cc:1467] Processing CreateTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 (DEFAULT_TABLE table=default.Table_dab2393d93b14ec79d96807a37360e33 [id=fcec4978e3584a9cabd5641b5ffb7340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.623080 1157 ts_tablet_manager.cc:1905] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:42.625705 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet be88a4cf79c14c0fb3ab1110a3fa1714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be88a4cf79c14c0fb3ab1110a3fa1714 already in progress: creating tablet
W20250109 20:52:42.626056 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ba365ba04f6848328a9b14dc4c92ac2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba365ba04f6848328a9b14dc4c92ac2d already in progress: creating tablet
W20250109 20:52:42.626307 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 899f08250f2248778f6abab551093461 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 899f08250f2248778f6abab551093461 already in progress: creating tablet
I20250109 20:52:42.626346 1165 tablet_bootstrap.cc:654] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.628391 893 tablet_service.cc:1467] Processing CreateTablet for tablet 6bea28fb446e49669133e21b02c416ad (DEFAULT_TABLE table=default.Table_401fcdac6f4d4150b7acedb7e3c8790e [id=06cecc207bb746dd9b0cb0f6eac4017e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.630017 896 tablet_service.cc:1467] Processing CreateTablet for tablet 58817dddbe8749249a96c60dee41b520 (DEFAULT_TABLE table=default.Table_38a61ab14f8c46f0b9bd21c3d95c9ac4 [id=fc46d1a61d7d4769bfcf249712d2e2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.632771 1157 ts_tablet_manager.cc:1918] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.632910 1157 log.cc:1198] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/8389e66e581f406bb82735a1ece78773
I20250109 20:52:42.635988 1157 ts_tablet_manager.cc:1939] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:42.638561 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 8389e66e581f406bb82735a1ece78773 (table default.Table_39dd6100bb4044f6a89213a6875f8756 [id=d641ab17c06741d4a0816205d1f4b16a]) successfully deleted
I20250109 20:52:42.641149 899 tablet_service.cc:1467] Processing CreateTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d (DEFAULT_TABLE table=default.Table_df0d03a66d1643bca34040c9aa2e1f3e [id=ae7fb9326c2c49a593d1111d45ea424f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.641397 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ec6b6b1ec1462d82008a78a885032d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.641579 885 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.641719 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.641865 887 tablet_service.cc:1514] Processing DeleteTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.642001 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 8389e66e581f406bb82735a1ece78773 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.642171 888 tablet_service.cc:1467] Processing CreateTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc (DEFAULT_TABLE table=default.Table_3eef1bf244124c8d8d9bda7d0699a6f7 [id=0048f4157ceb4c098d924be9004eeaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.642345 1165 log.cc:826] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Log is configured to *not* fsync() on all Append() calls
I20250109 20:52:42.643157 890 tablet_service.cc:1467] Processing CreateTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d (DEFAULT_TABLE table=default.Table_8c0e3ac68686429dadf4fcc158cf285b [id=0acf665b047a4f1dac232ef98359b372]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.645934 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a1a6a0a1f4411aa0b57481b6291808. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:42.652738 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6c38c8f0299e49ff8aff99b17b7be01b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c38c8f0299e49ff8aff99b17b7be01b already in progress: creating tablet
W20250109 20:52:42.653326 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 103722f5a2314bbeb9b141bb0a460ece because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 103722f5a2314bbeb9b141bb0a460ece already in progress: creating tablet
W20250109 20:52:42.653560 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ba365ba04f6848328a9b14dc4c92ac2d because the tablet was not found. No further retry: Not found: Tablet not found: ba365ba04f6848328a9b14dc4c92ac2d
W20250109 20:52:42.653739 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 899f08250f2248778f6abab551093461 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 899f08250f2248778f6abab551093461 already in progress: creating tablet
I20250109 20:52:42.653846 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58817dddbe8749249a96c60dee41b520. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:42.653977 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 920a6be9db344c098c62bfd2fa5a73e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 920a6be9db344c098c62bfd2fa5a73e8 already in progress: creating tablet
W20250109 20:52:42.655043 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 75ec6b6b1ec1462d82008a78a885032d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ec6b6b1ec1462d82008a78a885032d already in progress: creating tablet
W20250109 20:52:42.655305 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ecee7292068c4d649a1232ac54dbb7f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecee7292068c4d649a1232ac54dbb7f0 already in progress: creating tablet
W20250109 20:52:42.655494 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f6a1a6a0a1f4411aa0b57481b6291808 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6a1a6a0a1f4411aa0b57481b6291808 already in progress: creating tablet
W20250109 20:52:42.655683 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet be88a4cf79c14c0fb3ab1110a3fa1714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be88a4cf79c14c0fb3ab1110a3fa1714 already in progress: creating tablet
W20250109 20:52:42.655908 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8389e66e581f406bb82735a1ece78773 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8389e66e581f406bb82735a1ece78773 already in progress: creating tablet
I20250109 20:52:42.659893 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bea28fb446e49669133e21b02c416ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.660177 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d6dc79bf1e48ee9cf2bbb70463741d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.663379 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f5b4af510942c8b50838df3bc8d8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.668174 1158 tablet_bootstrap.cc:492] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:42.668332 1158 ts_tablet_manager.cc:1397] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.119s user 0.004s sys 0.019s
I20250109 20:52:42.669088 1158 raft_consensus.cc:357] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.669296 1158 raft_consensus.cc:383] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.669376 1158 raft_consensus.cc:738] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.669646 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.669620 1158 consensus_queue.cc:260] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.670114 1158 ts_tablet_manager.cc:1428] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250109 20:52:42.670327 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f6a1a6a0a1f4411aa0b57481b6291808 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6a1a6a0a1f4411aa0b57481b6291808 already in progress: creating tablet
I20250109 20:52:42.670635 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:42.671182 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ecee7292068c4d649a1232ac54dbb7f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecee7292068c4d649a1232ac54dbb7f0 already in progress: creating tablet
I20250109 20:52:42.672853 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 8389e66e581f406bb82735a1ece78773 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:42.673413 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8389e66e581f406bb82735a1ece78773 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8389e66e581f406bb82735a1ece78773 already in progress: creating tablet
I20250109 20:52:42.670663 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8389e66e581f406bb82735a1ece78773" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:42.674387 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e06f920c44304e0c806a3aa165fde5d0" } new_table_name: "default.Table_d4f1e00e338345deb365b807a7e0dddf" modify_external_catalogs: true
W20250109 20:52:42.674468 713 leader_election.cc:343] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:42.676906 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8389e66e581f406bb82735a1ece78773 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.677484 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 103722f5a2314bbeb9b141bb0a460ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.678284 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2aacfc7eea3490dbeb9d5c82d48766d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.682901 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_401fcdac6f4d4150b7acedb7e3c8790e" } new_table_name: "default.Table_8e9a23e5c7864ef78f06146e2469dc3c" modify_external_catalogs: true
I20250109 20:52:42.682924 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1f430949a522426db107b249ce1ab5f0" } new_table_name: "default.Table_82b978780ede49bfaa599c5af6e7d57d" modify_external_catalogs: true
I20250109 20:52:42.687947 1176 tablet_bootstrap.cc:492] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:42.688220 1176 ts_tablet_manager.cc:1397] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.161s user 0.012s sys 0.013s
I20250109 20:52:42.686002 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8389e66e581f406bb82735a1ece78773" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:42.691658 1165 tablet_bootstrap.cc:492] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:42.691929 1165 ts_tablet_manager.cc:1397] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.266s user 0.010s sys 0.000s
I20250109 20:52:42.692245 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 75ec6b6b1ec1462d82008a78a885032d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:42.692561 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 0454b27918ab41ad9ab269185b2692fe from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:42.692953 454 master_service.cc:946] ReplaceTablet: received request to replace tablet ecee7292068c4d649a1232ac54dbb7f0 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:42.693364 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 8389e66e581f406bb82735a1ece78773 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:42.693845 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_38a61ab14f8c46f0b9bd21c3d95c9ac4" } new_table_name: "default.Table_68ea6c2a7f6e4abb9728c2ab20e3b17e" modify_external_catalogs: true
W20250109 20:52:42.694108 712 leader_election.cc:343] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:42.694366 712 leader_election.cc:304] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:52:42.694814 712 raft_consensus.cc:2681] T 8389e66e581f406bb82735a1ece78773 P 09833ea46b7141d58c3899dace69e49f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:52:42.694764 1176 raft_consensus.cc:357] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.695106 1176 raft_consensus.cc:383] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.695194 1176 raft_consensus.cc:738] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.695345 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_81bc625ee9694d2ba87e1dafcbcad1f9" } new_table_name: "default.Table_e6d0cb94669f451389e1f2b2581c63b6" modify_external_catalogs: true
I20250109 20:52:42.695873 1176 consensus_queue.cc:260] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.692248 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 920a6be9db344c098c62bfd2fa5a73e8 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:42.697793 890 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 (DEFAULT_TABLE table=default.Table_7cbb1b8e12d44f4198d361e804254e84 [id=6af19c1506c9416d979c0d938080d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.698109 1157 tablet_replica.cc:331] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:42.698185 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fb6aeedc77452f99c46f03d6d8d089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.698309 1157 raft_consensus.cc:2238] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.698477 1157 raft_consensus.cc:2267] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:42.698536 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8389e66e581f406bb82735a1ece78773 because the tablet was not found. No further retry: Not found: Tablet not found: 8389e66e581f406bb82735a1ece78773
W20250109 20:52:42.698984 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdd067c335354357b44dac433f8c1523 consensus state has no leader
I20250109 20:52:42.699297 1157 ts_tablet_manager.cc:1905] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.699553 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 (DEFAULT_TABLE table=default.Table_7cbb1b8e12d44f4198d361e804254e84 [id=6af19c1506c9416d979c0d938080d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.701085 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fb6aeedc77452f99c46f03d6d8d089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.701833 754 tablet_service.cc:1514] Processing DeleteTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.702219 753 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 (DEFAULT_TABLE table=default.Table_7cbb1b8e12d44f4198d361e804254e84 [id=6af19c1506c9416d979c0d938080d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.702579 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fb6aeedc77452f99c46f03d6d8d089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.705773 1157 ts_tablet_manager.cc:1918] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.705910 1157 log.cc:1198] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/103722f5a2314bbeb9b141bb0a460ece
I20250109 20:52:42.706235 1157 ts_tablet_manager.cc:1939] T 103722f5a2314bbeb9b141bb0a460ece P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:42.707265 1157 tablet_replica.cc:331] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:42.707414 1157 raft_consensus.cc:2238] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.707583 1157 raft_consensus.cc:2267] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.707993 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 103722f5a2314bbeb9b141bb0a460ece (table default.Table_0710fdbfdd7a4e7b892b501e77fc2e72 [id=606e0f04684243e8a30813edb6e96766]) successfully deleted
I20250109 20:52:42.709376 1158 tablet_bootstrap.cc:492] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:42.711061 1158 tablet_bootstrap.cc:654] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.715622 1176 ts_tablet_manager.cc:1428] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.027s user 0.013s sys 0.006s
I20250109 20:52:42.716244 1165 raft_consensus.cc:357] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.716653 1165 raft_consensus.cc:383] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.716790 1165 raft_consensus.cc:738] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.717125 1176 tablet_bootstrap.cc:492] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:42.717343 1157 ts_tablet_manager.cc:1905] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.719177 1176 tablet_bootstrap.cc:654] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.719724 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 899f08250f2248778f6abab551093461 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.720331 1166 tablet_replica.cc:331] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:42.720829 1166 raft_consensus.cc:2238] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.721272 1166 raft_consensus.cc:2267] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.723304 1166 ts_tablet_manager.cc:1905] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.724882 1165 consensus_queue.cc:260] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.728345 1157 ts_tablet_manager.cc:1918] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.728466 1157 log.cc:1198] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/be88a4cf79c14c0fb3ab1110a3fa1714
I20250109 20:52:42.728615 1165 ts_tablet_manager.cc:1428] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.037s user 0.007s sys 0.004s
I20250109 20:52:42.732362 1166 ts_tablet_manager.cc:1918] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.732614 1166 log.cc:1198] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/899f08250f2248778f6abab551093461
I20250109 20:52:42.733237 1166 ts_tablet_manager.cc:1939] T 899f08250f2248778f6abab551093461 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:42.734073 1158 tablet_bootstrap.cc:492] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:42.734221 1158 ts_tablet_manager.cc:1397] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250109 20:52:42.735096 1158 raft_consensus.cc:357] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.735301 1158 raft_consensus.cc:383] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.735374 1158 raft_consensus.cc:738] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.735566 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 899f08250f2248778f6abab551093461 (table default.Table_732fb1f4c69449e8950e5fa85c7a2948 [id=a667d9b2080d44b89a1715b9b57c8e8b]) successfully deleted
I20250109 20:52:42.735623 1158 consensus_queue.cc:260] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.736668 1158 ts_tablet_manager.cc:1428] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:42.736934 1176 tablet_bootstrap.cc:492] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:42.736997 1158 tablet_bootstrap.cc:492] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:42.737054 1176 ts_tablet_manager.cc:1397] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.009s
I20250109 20:52:42.737499 1165 tablet_bootstrap.cc:492] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:42.737738 1176 raft_consensus.cc:357] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.737901 1176 raft_consensus.cc:383] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.737988 1176 raft_consensus.cc:738] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.738204 1176 consensus_queue.cc:260] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.738523 1158 tablet_bootstrap.cc:654] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.739280 1165 tablet_bootstrap.cc:654] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.739321 1176 ts_tablet_manager.cc:1428] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:42.739564 1176 tablet_bootstrap.cc:492] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:42.741438 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 8389e66e581f406bb82735a1ece78773 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.741724 1166 tablet_replica.cc:331] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:42.741948 1166 raft_consensus.cc:2238] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.742126 1166 raft_consensus.cc:2267] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.742134 1176 tablet_bootstrap.cc:654] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.742748 1166 ts_tablet_manager.cc:1905] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.746202 1158 tablet_bootstrap.cc:492] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:42.746349 1158 ts_tablet_manager.cc:1397] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20250109 20:52:42.747035 1158 raft_consensus.cc:357] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.747262 1158 raft_consensus.cc:383] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.748808 1157 ts_tablet_manager.cc:1939] T be88a4cf79c14c0fb3ab1110a3fa1714 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:52:42.749892 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bea28fb446e49669133e21b02c416ad consensus state has no leader
I20250109 20:52:42.750914 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet be88a4cf79c14c0fb3ab1110a3fa1714 (table default.Table_1b67d2a3c0b3403b8041cfb32c37b227 [id=e68ca2b1501a493a97b5ae61bdd9fe60]) successfully deleted
I20250109 20:52:42.752497 1166 ts_tablet_manager.cc:1918] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.752748 1166 log.cc:1198] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/8389e66e581f406bb82735a1ece78773
I20250109 20:52:42.753286 1166 ts_tablet_manager.cc:1939] T 8389e66e581f406bb82735a1ece78773 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:42.754386 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 8389e66e581f406bb82735a1ece78773 (table default.Table_39dd6100bb4044f6a89213a6875f8756 [id=d641ab17c06741d4a0816205d1f4b16a]) successfully deleted
I20250109 20:52:42.754639 1176 tablet_bootstrap.cc:492] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:42.754787 1176 ts_tablet_manager.cc:1397] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20250109 20:52:42.755252 1176 raft_consensus.cc:357] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.755848 1176 raft_consensus.cc:383] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.755916 1176 raft_consensus.cc:738] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.756119 1176 consensus_queue.cc:260] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.757022 1165 tablet_bootstrap.cc:492] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:42.757191 1165 ts_tablet_manager.cc:1397] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20250109 20:52:42.747359 1158 raft_consensus.cc:738] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.757625 1176 ts_tablet_manager.cc:1428] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:42.757879 1176 tablet_bootstrap.cc:492] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:42.757807 1158 consensus_queue.cc:260] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.759996 1176 tablet_bootstrap.cc:654] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:42.760540 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58817dddbe8749249a96c60dee41b520 consensus state has no leader
W20250109 20:52:42.760998 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38760b86fdfb4c8c8bcfd90874cbfd0d consensus state has no leader
I20250109 20:52:42.761683 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_1f430949a522426db107b249ce1ab5f0 [id=e4a51641b240493d9f69b9031ef5d4b1]). Expected version 1 got 0
W20250109 20:52:42.762086 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e784a5378f7740baa39673071b56d8c5 consensus state has no leader
I20250109 20:52:42.765192 1158 ts_tablet_manager.cc:1428] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250109 20:52:42.766309 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 30ae6c4e835048bd8deed5ac9c4ae07a (table default.Table_a6d4d40a237548c384fd897eb207903c [id=6fe0e6310b9f44a8b3b9558fec22c2a6]). Expected version 1 got 0
I20250109 20:52:42.767525 1158 tablet_bootstrap.cc:492] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:42.769163 1158 tablet_bootstrap.cc:654] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.769196 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.770145 1176 tablet_bootstrap.cc:492] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:42.770275 1176 ts_tablet_manager.cc:1397] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20250109 20:52:42.771428 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 34cfa3005fdd40709a630d5bb80564db (DEFAULT_TABLE table=default.Table_5c35bca9b68e42bb9075f51687f74d15 [id=261eb60785fc4d11909ad99fe7cb53d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.771754 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cfa3005fdd40709a630d5bb80564db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.771677 1176 raft_consensus.cc:357] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.771842 1176 raft_consensus.cc:383] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.771911 1176 raft_consensus.cc:738] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.772121 1176 consensus_queue.cc:260] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.772951 1020 tablet_service.cc:1467] Processing CreateTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe (DEFAULT_TABLE table=default.Table_e1e2df6604b3481583f4036b3aee752e [id=ae10da31c7cf49789ddf4c05c169c105]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.773276 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeedf2e359c94a96ba6a518cefbb1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.773710 753 tablet_service.cc:1467] Processing CreateTablet for tablet bd445a1adba34529aa3f7a44b991b177 (DEFAULT_TABLE table=default.Table_9af6ca0c3f0042358b13f0891ced6dea [id=16fc7304546a423abcbcd230d06efd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.774170 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd445a1adba34529aa3f7a44b991b177. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.772679 1176 ts_tablet_manager.cc:1428] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:42.775077 1176 tablet_bootstrap.cc:492] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:42.776422 1176 tablet_bootstrap.cc:654] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.776984 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 660e60f368cb4e25b12ab3d7b2e314b1 (DEFAULT_TABLE table=default.Table_709427497f3a436a99bee6129d87efb6 [id=028851fe80724d3cb4402101784bd12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.777030 754 tablet_service.cc:1467] Processing CreateTablet for tablet 34cfa3005fdd40709a630d5bb80564db (DEFAULT_TABLE table=default.Table_5c35bca9b68e42bb9075f51687f74d15 [id=261eb60785fc4d11909ad99fe7cb53d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.765761 890 tablet_service.cc:1467] Processing CreateTablet for tablet bd445a1adba34529aa3f7a44b991b177 (DEFAULT_TABLE table=default.Table_9af6ca0c3f0042358b13f0891ced6dea [id=16fc7304546a423abcbcd230d06efd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.777261 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660e60f368cb4e25b12ab3d7b2e314b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.777395 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cfa3005fdd40709a630d5bb80564db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.777447 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd445a1adba34529aa3f7a44b991b177. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.780964 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ab00e387fbc843b2841bef2d3289df34 (DEFAULT_TABLE table=default.Table_49a9931b8903487db0856c10d60fc1e9 [id=8e13eabc06da4067aa8108f30c8d323d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.781001 757 tablet_service.cc:1467] Processing CreateTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe (DEFAULT_TABLE table=default.Table_e1e2df6604b3481583f4036b3aee752e [id=ae10da31c7cf49789ddf4c05c169c105]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.781232 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab00e387fbc843b2841bef2d3289df34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.781380 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeedf2e359c94a96ba6a518cefbb1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.782542 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 30ae6c4e835048bd8deed5ac9c4ae07a (table default.Table_a6d4d40a237548c384fd897eb207903c [id=6fe0e6310b9f44a8b3b9558fec22c2a6]). Expected version 1 got 0
I20250109 20:52:42.766139 895 tablet_service.cc:1467] Processing CreateTablet for tablet 34cfa3005fdd40709a630d5bb80564db (DEFAULT_TABLE table=default.Table_5c35bca9b68e42bb9075f51687f74d15 [id=261eb60785fc4d11909ad99fe7cb53d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.783605 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cfa3005fdd40709a630d5bb80564db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.784982 751 tablet_service.cc:1467] Processing CreateTablet for tablet 660e60f368cb4e25b12ab3d7b2e314b1 (DEFAULT_TABLE table=default.Table_709427497f3a436a99bee6129d87efb6 [id=028851fe80724d3cb4402101784bd12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.785611 1033 tablet_service.cc:1467] Processing CreateTablet for tablet bd445a1adba34529aa3f7a44b991b177 (DEFAULT_TABLE table=default.Table_9af6ca0c3f0042358b13f0891ced6dea [id=16fc7304546a423abcbcd230d06efd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.785880 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd445a1adba34529aa3f7a44b991b177. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.786036 752 tablet_service.cc:1467] Processing CreateTablet for tablet ab00e387fbc843b2841bef2d3289df34 (DEFAULT_TABLE table=default.Table_49a9931b8903487db0856c10d60fc1e9 [id=8e13eabc06da4067aa8108f30c8d323d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.786422 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab00e387fbc843b2841bef2d3289df34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.766490 893 tablet_service.cc:1467] Processing CreateTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe (DEFAULT_TABLE table=default.Table_e1e2df6604b3481583f4036b3aee752e [id=ae10da31c7cf49789ddf4c05c169c105]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.766889 899 tablet_service.cc:1467] Processing CreateTablet for tablet 660e60f368cb4e25b12ab3d7b2e314b1 (DEFAULT_TABLE table=default.Table_709427497f3a436a99bee6129d87efb6 [id=028851fe80724d3cb4402101784bd12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.767179 888 tablet_service.cc:1467] Processing CreateTablet for tablet ab00e387fbc843b2841bef2d3289df34 (DEFAULT_TABLE table=default.Table_49a9931b8903487db0856c10d60fc1e9 [id=8e13eabc06da4067aa8108f30c8d323d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.788645 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_130e9f2c3d5c42b8b1147cbf8a482014" } modify_external_catalogs: true
I20250109 20:52:42.788925 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660e60f368cb4e25b12ab3d7b2e314b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.788858 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a6d4d40a237548c384fd897eb207903c" } modify_external_catalogs: true
I20250109 20:52:42.788910 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_130e9f2c3d5c42b8b1147cbf8a482014" } modify_external_catalogs: true
I20250109 20:52:42.789026 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a6d4d40a237548c384fd897eb207903c" } modify_external_catalogs: true
I20250109 20:52:42.792131 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeedf2e359c94a96ba6a518cefbb1fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.792613 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9e3ee29227aa4e07b4f331683046e066" } modify_external_catalogs: true
I20250109 20:52:42.792769 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9e3ee29227aa4e07b4f331683046e066" } modify_external_catalogs: true
I20250109 20:52:42.796594 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e1e2df6604b3481583f4036b3aee752e" } modify_external_catalogs: true
I20250109 20:52:42.796929 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab00e387fbc843b2841bef2d3289df34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.796988 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e1e2df6604b3481583f4036b3aee752e" } modify_external_catalogs: true
I20250109 20:52:42.801067 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660e60f368cb4e25b12ab3d7b2e314b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.801326 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 68201559ee26497cb974891a5917e023
I20250109 20:52:42.802433 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0454b27918ab41ad9ab269185b2692fe
I20250109 20:52:42.802847 447 catalog_manager.cc:6596] ReplaceTablet: tablet 0454b27918ab41ad9ab269185b2692fe deleted and replaced by tablet 35d7cead4f4149ae8de48eee2cfd8679
I20250109 20:52:42.804782 750 tablet_service.cc:1514] Processing DeleteTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.804998 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454b27918ab41ad9ab269185b2692fe with delete_type TABLET_DATA_DELETED (replaced by tablet 35d7cead4f4149ae8de48eee2cfd8679) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.805025 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 30ae6c4e835048bd8deed5ac9c4ae07a
I20250109 20:52:42.805006 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454b27918ab41ad9ab269185b2692fe with delete_type TABLET_DATA_DELETED (replaced by tablet 35d7cead4f4149ae8de48eee2cfd8679) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.805181 1157 tablet_replica.cc:331] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:42.805330 1157 raft_consensus.cc:2238] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.805483 1157 raft_consensus.cc:2267] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:42.805742 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0454b27918ab41ad9ab269185b2692fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0454b27918ab41ad9ab269185b2692fe already in progress: creating tablet
I20250109 20:52:42.805948 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ae6c4e835048bd8deed5ac9c4ae07a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.806152 1157 ts_tablet_manager.cc:1905] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.806311 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ae6c4e835048bd8deed5ac9c4ae07a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:42.806499 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 30ae6c4e835048bd8deed5ac9c4ae07a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ae6c4e835048bd8deed5ac9c4ae07a already in progress: creating tablet
W20250109 20:52:42.808037 971 tablet.cc:2367] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:52:42.811269 1165 raft_consensus.cc:357] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.811439 1165 raft_consensus.cc:383] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.811522 1165 raft_consensus.cc:738] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.811729 1165 consensus_queue.cc:260] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.812201 1165 ts_tablet_manager.cc:1428] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:42.812295 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ae6c4e835048bd8deed5ac9c4ae07a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.816776 1165 tablet_bootstrap.cc:492] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:42.818225 1165 tablet_bootstrap.cc:654] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.818722 1157 ts_tablet_manager.cc:1918] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.819352 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454b27918ab41ad9ab269185b2692fe with delete_type TABLET_DATA_DELETED (replaced by tablet 35d7cead4f4149ae8de48eee2cfd8679) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.819782 1158 tablet_bootstrap.cc:492] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:42.819911 1158 ts_tablet_manager.cc:1397] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.021s
I20250109 20:52:42.819420 1157 log.cc:1198] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ba365ba04f6848328a9b14dc4c92ac2d
I20250109 20:52:42.820420 1157 ts_tablet_manager.cc:1939] T ba365ba04f6848328a9b14dc4c92ac2d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:42.820662 1158 raft_consensus.cc:357] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.820832 1158 raft_consensus.cc:383] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.820930 1158 raft_consensus.cc:738] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.821123 1158 consensus_queue.cc:260] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.821525 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ba365ba04f6848328a9b14dc4c92ac2d (table default.Table_2e5f440a1e9a4e6083d0857464311f92 [id=613376bfcfe64a468bcc9dc242929efe]) successfully deleted
I20250109 20:52:42.822124 1158 ts_tablet_manager.cc:1428] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:42.822346 1158 tablet_bootstrap.cc:492] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:42.824488 1158 tablet_bootstrap.cc:654] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.824505 1166 tablet_replica.cc:331] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:42.824725 1166 raft_consensus.cc:2238] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:42.825030 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0454b27918ab41ad9ab269185b2692fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0454b27918ab41ad9ab269185b2692fe already in progress: creating tablet
I20250109 20:52:42.825364 1157 tablet_replica.cc:331] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:42.825414 1166 raft_consensus.cc:2267] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.825573 1157 raft_consensus.cc:2238] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.825758 1157 raft_consensus.cc:2267] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.826514 1157 ts_tablet_manager.cc:1905] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.827270 1166 ts_tablet_manager.cc:1905] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.832625 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.832921 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_82b978780ede49bfaa599c5af6e7d57d [id=e4a51641b240493d9f69b9031ef5d4b1]). Expected version 2 got 0
I20250109 20:52:42.833640 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3eef1bf244124c8d8d9bda7d0699a6f7" } modify_external_catalogs: true
I20250109 20:52:42.833840 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3eef1bf244124c8d8d9bda7d0699a6f7" } modify_external_catalogs: true
I20250109 20:52:42.836196 1158 tablet_bootstrap.cc:492] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:42.836364 1158 ts_tablet_manager.cc:1397] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20250109 20:52:42.836752 1166 ts_tablet_manager.cc:1918] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.836858 1166 log.cc:1198] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ba365ba04f6848328a9b14dc4c92ac2d
I20250109 20:52:42.837008 1158 raft_consensus.cc:357] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.837170 1158 raft_consensus.cc:383] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.837218 1158 raft_consensus.cc:738] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.837284 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 7fc0cbb582d84b579d1a32909c2512c7
I20250109 20:52:42.837424 1158 consensus_queue.cc:260] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.837908 1158 ts_tablet_manager.cc:1428] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:52:42.838122 1158 tablet_bootstrap.cc:492] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:42.838548 1166 ts_tablet_manager.cc:1939] T ba365ba04f6848328a9b14dc4c92ac2d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:42.839363 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ba365ba04f6848328a9b14dc4c92ac2d (table default.Table_2e5f440a1e9a4e6083d0857464311f92 [id=613376bfcfe64a468bcc9dc242929efe]) successfully deleted
I20250109 20:52:42.839399 1158 tablet_bootstrap.cc:654] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:42.840854 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 30ae6c4e835048bd8deed5ac9c4ae07a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30ae6c4e835048bd8deed5ac9c4ae07a already in progress: creating tablet
W20250109 20:52:42.841677 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0454b27918ab41ad9ab269185b2692fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0454b27918ab41ad9ab269185b2692fe already in progress: creating tablet
I20250109 20:52:42.842867 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.849584 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet eeedf2e359c94a96ba6a518cefbb1fbe
I20250109 20:52:42.849715 1165 tablet_bootstrap.cc:492] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:42.849901 1165 ts_tablet_manager.cc:1397] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20250109 20:52:42.850600 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.850834 895 tablet_service.cc:1514] Processing DeleteTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.850793 1165 raft_consensus.cc:357] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.851053 1165 raft_consensus.cc:383] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.851135 1165 raft_consensus.cc:738] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:52:42.851526 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet eeedf2e359c94a96ba6a518cefbb1fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeedf2e359c94a96ba6a518cefbb1fbe already in progress: creating tablet
I20250109 20:52:42.851397 1165 consensus_queue.cc:260] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:42.851910 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet eeedf2e359c94a96ba6a518cefbb1fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeedf2e359c94a96ba6a518cefbb1fbe already in progress: creating tablet
I20250109 20:52:42.851984 1165 ts_tablet_manager.cc:1428] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:42.852355 753 tablet_service.cc:1514] Processing DeleteTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.856173 1165 tablet_bootstrap.cc:492] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:42.858103 1165 tablet_bootstrap.cc:654] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.859129 754 tablet_service.cc:1467] Processing CreateTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 (DEFAULT_TABLE table=default.Table_60febb828b3c4979bf94fd0f01bdfe4c [id=4e6d2e7acdc84a8fa2d3bd56e8437974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.859449 895 tablet_service.cc:1467] Processing CreateTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 (DEFAULT_TABLE table=default.Table_02728ead2eff4bd4b7809e4da74fe753 [id=53044a76783c48ed945a98bfc0bfb90c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.859539 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68aa4a05c36e44b0aa6ee03780d2a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.859777 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded8e3a447c84b9b8ec4ccd70199ccd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.862154 1157 ts_tablet_manager.cc:1918] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.862257 1157 log.cc:1198] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/30ae6c4e835048bd8deed5ac9c4ae07a
I20250109 20:52:42.862565 1157 ts_tablet_manager.cc:1939] T 30ae6c4e835048bd8deed5ac9c4ae07a P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:42.863498 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 30ae6c4e835048bd8deed5ac9c4ae07a (table default.Table_a6d4d40a237548c384fd897eb207903c [id=6fe0e6310b9f44a8b3b9558fec22c2a6]) successfully deleted
I20250109 20:52:42.864887 893 tablet_service.cc:1467] Processing CreateTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 (DEFAULT_TABLE table=default.Table_60febb828b3c4979bf94fd0f01bdfe4c [id=4e6d2e7acdc84a8fa2d3bd56e8437974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.865180 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68aa4a05c36e44b0aa6ee03780d2a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.866829 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 26f5b4af510942c8b50838df3bc8d8fc
I20250109 20:52:42.868227 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:42.868822 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454b27918ab41ad9ab269185b2692fe with delete_type TABLET_DATA_DELETED (replaced by tablet 35d7cead4f4149ae8de48eee2cfd8679) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.868796 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.869199 753 tablet_service.cc:1467] Processing CreateTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 (DEFAULT_TABLE table=default.Table_02728ead2eff4bd4b7809e4da74fe753 [id=53044a76783c48ed945a98bfc0bfb90c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:42.869341 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 26f5b4af510942c8b50838df3bc8d8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26f5b4af510942c8b50838df3bc8d8fc already in progress: creating tablet
I20250109 20:52:42.869524 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded8e3a447c84b9b8ec4ccd70199ccd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.869987 1157 tablet_replica.cc:331] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:42.870167 1157 raft_consensus.cc:2238] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.870314 1157 raft_consensus.cc:2267] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.873050 1160 tablet_replica.cc:331] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:42.873499 1160 raft_consensus.cc:2238] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.873886 1160 raft_consensus.cc:2267] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.876416 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.877074 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.878461 1160 ts_tablet_manager.cc:1905] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.878577 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 (DEFAULT_TABLE table=default.Table_60febb828b3c4979bf94fd0f01bdfe4c [id=4e6d2e7acdc84a8fa2d3bd56e8437974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.878940 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68aa4a05c36e44b0aa6ee03780d2a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.879909 1020 tablet_service.cc:1467] Processing CreateTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 (DEFAULT_TABLE table=default.Table_02728ead2eff4bd4b7809e4da74fe753 [id=53044a76783c48ed945a98bfc0bfb90c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.880227 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded8e3a447c84b9b8ec4ccd70199ccd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.882057 1165 tablet_bootstrap.cc:492] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:42.882213 1165 ts_tablet_manager.cc:1397] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20250109 20:52:42.883240 1165 raft_consensus.cc:357] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.883486 1165 raft_consensus.cc:383] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.883587 1165 raft_consensus.cc:738] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.883906 1165 consensus_queue.cc:260] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.884111 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e440d446aaba4d62a8ea1ad840f9147c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.884596 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e440d446aaba4d62a8ea1ad840f9147c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.886673 754 tablet_service.cc:1467] Processing CreateTablet for tablet 35d7cead4f4149ae8de48eee2cfd8679 (DEFAULT_TABLE table=default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.886942 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 35d7cead4f4149ae8de48eee2cfd8679 (DEFAULT_TABLE table=default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.887049 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d7cead4f4149ae8de48eee2cfd8679. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.887223 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d7cead4f4149ae8de48eee2cfd8679. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.888938 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9785d67f4597409cb3bb0414d0166b4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.889137 1165 ts_tablet_manager.cc:1428] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
W20250109 20:52:42.889367 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9785d67f4597409cb3bb0414d0166b4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.890801 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.892928 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_49d1907147ef4990876347467005e31e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.893378 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49d1907147ef4990876347467005e31e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.899235 893 tablet_service.cc:1467] Processing CreateTablet for tablet 35d7cead4f4149ae8de48eee2cfd8679 (DEFAULT_TABLE table=default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.899614 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d7cead4f4149ae8de48eee2cfd8679. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.900700 1157 ts_tablet_manager.cc:1905] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.905169 1176 tablet_bootstrap.cc:492] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:42.905329 1176 ts_tablet_manager.cc:1397] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.130s user 0.020s sys 0.009s
I20250109 20:52:42.906044 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b2dd9dd1ca5c4c1b9d2925b5110c81e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.906081 1176 raft_consensus.cc:357] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.906258 1176 raft_consensus.cc:383] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.906952 1176 raft_consensus.cc:738] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.906988 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1bca442744ee4e04a46973fb21434cae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.907922 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7cbb1b8e12d44f4198d361e804254e84" } new_table_name: "default.Table_d47b6907b4ec4c8bbd9b27d5a1dc6be5" modify_external_catalogs: true
I20250109 20:52:42.909363 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e2cedbf0f0cd4d88ae098ddd22cb9972"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.909793 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2cedbf0f0cd4d88ae098ddd22cb9972 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.910979 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c576a7a39c174fe6ac5c351e1b4f80d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.911347 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c576a7a39c174fe6ac5c351e1b4f80d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:42.911463 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bca442744ee4e04a46973fb21434cae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.911505 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f580256f4281489b85b9f46fcd381c5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.912150 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a58d6a9aabde40128ad0585f772ef591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:42.912515 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a58d6a9aabde40128ad0585f772ef591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:42.912901 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f580256f4281489b85b9f46fcd381c5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.912878 1176 consensus_queue.cc:260] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.914846 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e0f7637d2d6c427a9d08b9e3e4f307e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:42.914999 1176 ts_tablet_manager.cc:1428] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250109 20:52:42.915315 1176 tablet_bootstrap.cc:492] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:42.915468 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0f7637d2d6c427a9d08b9e3e4f307e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.917443 1176 tablet_bootstrap.cc:654] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:42.917718 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2dd9dd1ca5c4c1b9d2925b5110c81e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:42.920315 1157 ts_tablet_manager.cc:1918] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.920451 1157 log.cc:1198] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6c38c8f0299e49ff8aff99b17b7be01b
I20250109 20:52:42.920931 1157 ts_tablet_manager.cc:1939] T 6c38c8f0299e49ff8aff99b17b7be01b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:42.921763 1160 ts_tablet_manager.cc:1918] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.921922 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6c38c8f0299e49ff8aff99b17b7be01b (table default.Table_70760925d6df4f0c961c825337270e1a [id=d9fb05f1ceef49638bbe6db78d2d17c2]) successfully deleted
I20250109 20:52:42.922302 1157 tablet_replica.cc:331] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:42.922497 1157 raft_consensus.cc:2238] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.922726 1157 raft_consensus.cc:2267] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.923171 1160 log.cc:1198] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/0454b27918ab41ad9ab269185b2692fe
I20250109 20:52:42.923492 1157 ts_tablet_manager.cc:1905] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.923705 1160 ts_tablet_manager.cc:1939] T 0454b27918ab41ad9ab269185b2692fe P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:42.925670 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 0454b27918ab41ad9ab269185b2692fe (table default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
I20250109 20:52:42.925758 1160 tablet_replica.cc:331] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:42.925946 1160 raft_consensus.cc:2238] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.926090 1160 raft_consensus.cc:2267] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:42.926177 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 26f5b4af510942c8b50838df3bc8d8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26f5b4af510942c8b50838df3bc8d8fc already in progress: creating tablet
I20250109 20:52:42.890841 1165 tablet_bootstrap.cc:492] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:42.927918 1176 tablet_bootstrap.cc:492] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:42.928097 1176 ts_tablet_manager.cc:1397] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20250109 20:52:42.928700 1165 tablet_bootstrap.cc:654] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.928969 1157 ts_tablet_manager.cc:1918] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.929081 1157 log.cc:1198] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/920a6be9db344c098c62bfd2fa5a73e8
I20250109 20:52:42.929013 1176 raft_consensus.cc:357] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.929464 1157 ts_tablet_manager.cc:1939] T 920a6be9db344c098c62bfd2fa5a73e8 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:42.929464 1176 raft_consensus.cc:383] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.929656 1176 raft_consensus.cc:738] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.929958 1176 consensus_queue.cc:260] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.930424 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 920a6be9db344c098c62bfd2fa5a73e8 (table default.Table_eb403509cf414f548af84d837a6f04ac [id=a1dea7259a764d7ca62d5ff76ccc95b8]) successfully deleted
W20250109 20:52:42.930819 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet eeedf2e359c94a96ba6a518cefbb1fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeedf2e359c94a96ba6a518cefbb1fbe already in progress: creating tablet
W20250109 20:52:42.931075 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 26f5b4af510942c8b50838df3bc8d8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26f5b4af510942c8b50838df3bc8d8fc already in progress: creating tablet
I20250109 20:52:42.931697 1176 ts_tablet_manager.cc:1428] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:42.931995 1176 tablet_bootstrap.cc:492] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:42.934017 1176 tablet_bootstrap.cc:654] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.936316 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 920a6be9db344c098c62bfd2fa5a73e8 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:42.937829 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 8389e66e581f406bb82735a1ece78773 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:42.938335 454 master_service.cc:946] ReplaceTablet: received request to replace tablet bd445a1adba34529aa3f7a44b991b177 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:42.935985 1165 tablet_bootstrap.cc:492] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:42.938673 1165 ts_tablet_manager.cc:1397] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20250109 20:52:42.939508 1165 raft_consensus.cc:357] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.939730 1165 raft_consensus.cc:383] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.939807 1165 raft_consensus.cc:738] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.940073 1165 consensus_queue.cc:260] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.940564 1176 tablet_bootstrap.cc:492] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:42.940776 1176 ts_tablet_manager.cc:1397] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250109 20:52:42.941485 1176 raft_consensus.cc:357] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.941743 1176 raft_consensus.cc:383] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:42.941861 1176 raft_consensus.cc:738] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:42.942147 1176 consensus_queue.cc:260] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:42.942548 1165 ts_tablet_manager.cc:1428] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:52:42.942806 1165 tablet_bootstrap.cc:492] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:42.943948 1176 ts_tablet_manager.cc:1428] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:52:42.944304 1176 tablet_bootstrap.cc:492] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:42.944895 1165 tablet_bootstrap.cc:654] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.945916 1176 tablet_bootstrap.cc:654] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:42.950083 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet cdd067c335354357b44dac433f8c1523 (table default.Table_d4f1e00e338345deb365b807a7e0dddf [id=8c860f8c96254692906f892a25530962]). Expected version 2 got 0
I20250109 20:52:42.950799 450 master_service.cc:946] ReplaceTablet: received request to replace tablet ba365ba04f6848328a9b14dc4c92ac2d from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:42.951843 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89fb6aeedc77452f99c46f03d6d8d089 consensus state has no leader
I20250109 20:52:42.954115 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a (DEFAULT_TABLE table=default.Table_9785d67f4597409cb3bb0414d0166b4c [id=d1bd5b484fbf40388a3a22838448be2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.954488 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a3dbc983534d65994fc0ef85d0dd2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.954630 753 tablet_service.cc:1467] Processing CreateTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a (DEFAULT_TABLE table=default.Table_9785d67f4597409cb3bb0414d0166b4c [id=d1bd5b484fbf40388a3a22838448be2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.954996 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a3dbc983534d65994fc0ef85d0dd2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.958321 1018 tablet_service.cc:1467] Processing CreateTablet for tablet ab593a602d61495eb7b323903ae610b6 (DEFAULT_TABLE table=default.Table_e440d446aaba4d62a8ea1ad840f9147c [id=df6e0f6dba1942d9b232a020d5d2b4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.958657 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab593a602d61495eb7b323903ae610b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.958721 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_02728ead2eff4bd4b7809e4da74fe753" } new_table_name: "default.Table_093ba0191f3b46f0aa7b6eb7c40f6f69" modify_external_catalogs: true
I20250109 20:52:42.954645 754 tablet_service.cc:1467] Processing CreateTablet for tablet ab593a602d61495eb7b323903ae610b6 (DEFAULT_TABLE table=default.Table_e440d446aaba4d62a8ea1ad840f9147c [id=df6e0f6dba1942d9b232a020d5d2b4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.961166 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab593a602d61495eb7b323903ae610b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.966799 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet cdd067c335354357b44dac433f8c1523 (table default.Table_d4f1e00e338345deb365b807a7e0dddf [id=8c860f8c96254692906f892a25530962]). Expected version 2 got 0
I20250109 20:52:42.968333 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.968581 1166 tablet_replica.cc:331] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:42.968792 1166 raft_consensus.cc:2238] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.968967 1166 raft_consensus.cc:2267] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.969637 1166 ts_tablet_manager.cc:1905] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.971491 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e0f7637d2d6c427a9d08b9e3e4f307e7" } new_table_name: "default.Table_d8c45162ae564eb0b502f22e5bfe50bd" modify_external_catalogs: true
I20250109 20:52:42.973691 895 tablet_service.cc:1467] Processing CreateTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a (DEFAULT_TABLE table=default.Table_9785d67f4597409cb3bb0414d0166b4c [id=d1bd5b484fbf40388a3a22838448be2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.973901 890 tablet_service.cc:1467] Processing CreateTablet for tablet ab593a602d61495eb7b323903ae610b6 (DEFAULT_TABLE table=default.Table_e440d446aaba4d62a8ea1ad840f9147c [id=df6e0f6dba1942d9b232a020d5d2b4d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:42.974196 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a3dbc983534d65994fc0ef85d0dd2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.971521 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d4f1e00e338345deb365b807a7e0dddf" } new_table_name: "default.Table_fd20b680db98408db7e7f0278832db97" modify_external_catalogs: true
I20250109 20:52:42.976413 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab593a602d61495eb7b323903ae610b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:42.982060 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bd445a1adba34529aa3f7a44b991b177
I20250109 20:52:42.982268 1166 ts_tablet_manager.cc:1918] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.982380 1166 log.cc:1198] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/be88a4cf79c14c0fb3ab1110a3fa1714
I20250109 20:52:42.982575 454 catalog_manager.cc:6596] ReplaceTablet: tablet bd445a1adba34529aa3f7a44b991b177 deleted and replaced by tablet 5496b2f965064f7788ebd74c62d1240a
I20250109 20:52:42.982666 1166 ts_tablet_manager.cc:1939] T be88a4cf79c14c0fb3ab1110a3fa1714 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:42.983237 899 tablet_service.cc:1514] Processing DeleteTablet for tablet bd445a1adba34529aa3f7a44b991b177 with delete_type TABLET_DATA_DELETED (replaced by tablet 5496b2f965064f7788ebd74c62d1240a) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:42.983785 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_df0d03a66d1643bca34040c9aa2e1f3e" } new_table_name: "default.Table_f9f28decdfb64455809f7b283552a4a5" modify_external_catalogs: true
I20250109 20:52:42.984555 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.985278 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet be88a4cf79c14c0fb3ab1110a3fa1714 (table default.Table_1b67d2a3c0b3403b8041cfb32c37b227 [id=e68ca2b1501a493a97b5ae61bdd9fe60]) successfully deleted
I20250109 20:52:42.985102 1166 tablet_replica.cc:331] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:42.985520 1166 raft_consensus.cc:2238] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:42.985672 1166 raft_consensus.cc:2267] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:42.985373 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet bd445a1adba34529aa3f7a44b991b177 with delete_type TABLET_DATA_DELETED (replaced by tablet 5496b2f965064f7788ebd74c62d1240a) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:42.986387 1166 ts_tablet_manager.cc:1905] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:42.986588 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 899f08250f2248778f6abab551093461 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:42.986979 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ded8e3a447c84b9b8ec4ccd70199ccd5 consensus state has no leader
I20250109 20:52:42.992012 1166 ts_tablet_manager.cc:1918] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:42.992100 1166 log.cc:1198] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ecee7292068c4d649a1232ac54dbb7f0
I20250109 20:52:42.992429 1166 ts_tablet_manager.cc:1939] T ecee7292068c4d649a1232ac54dbb7f0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:42.994431 1104 tablet.cc:2367] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:52:42.995643 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ecee7292068c4d649a1232ac54dbb7f0 (table default.Table_7a8859a022a54a48affdc0ef3024c194 [id=315670c6ced1467e82fc224f50ca5625]) successfully deleted
W20250109 20:52:42.996630 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet bd445a1adba34529aa3f7a44b991b177 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd445a1adba34529aa3f7a44b991b177 already in progress: creating tablet
W20250109 20:52:42.996786 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f5a90e65e80494a8b451f2bc6baa2b1 has no consensus state
W20250109 20:52:42.998240 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdd067c335354357b44dac433f8c1523 consensus state has no leader
I20250109 20:52:42.999058 1191 raft_consensus.cc:491] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:42.999298 1191 raft_consensus.cc:513] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.001291 1191 leader_election.cc:290] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:43.001981 1158 tablet_bootstrap.cc:492] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.004254 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c576a7a39c174fe6ac5c351e1b4f80d3" } modify_external_catalogs: true
I20250109 20:52:43.004480 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c576a7a39c174fe6ac5c351e1b4f80d3" } modify_external_catalogs: true
I20250109 20:52:43.004807 1158 ts_tablet_manager.cc:1397] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.167s user 0.061s sys 0.081s
W20250109 20:52:43.005173 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5d6dc79bf1e48ee9cf2bbb70463741d consensus state has no leader
I20250109 20:52:43.005841 1158 raft_consensus.cc:357] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.006028 1158 raft_consensus.cc:383] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.006410 1158 raft_consensus.cc:738] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.006489 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9785d67f4597409cb3bb0414d0166b4c" } modify_external_catalogs: true
I20250109 20:52:43.006672 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9785d67f4597409cb3bb0414d0166b4c" } modify_external_catalogs: true
I20250109 20:52:43.006655 1158 consensus_queue.cc:260] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.007141 1158 ts_tablet_manager.cc:1428] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:43.007370 1158 tablet_bootstrap.cc:492] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.009359 1158 tablet_bootstrap.cc:654] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.009999 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 26f5b4af510942c8b50838df3bc8d8fc from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.017392 1158 tablet_bootstrap.cc:492] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.017577 1158 ts_tablet_manager.cc:1397] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250109 20:52:43.018296 1158 raft_consensus.cc:357] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.018560 1158 raft_consensus.cc:383] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.018605 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]). Expected version 1 got 0
I20250109 20:52:43.018667 1158 raft_consensus.cc:738] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.018756 1160 ts_tablet_manager.cc:1905] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.019032 1158 consensus_queue.cc:260] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.019577 1158 ts_tablet_manager.cc:1428] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:43.019891 1158 tablet_bootstrap.cc:492] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.021328 1158 tablet_bootstrap.cc:654] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.024623 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 899f08250f2248778f6abab551093461 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.024848 1157 tablet_replica.cc:331] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:43.025000 1157 raft_consensus.cc:2238] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.025128 1157 raft_consensus.cc:2267] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.026361 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c6a3dbc983534d65994fc0ef85d0dd2a
I20250109 20:52:43.027225 752 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.027287 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.027916 1160 ts_tablet_manager.cc:1918] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:43.027981 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c6a3dbc983534d65994fc0ef85d0dd2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a3dbc983534d65994fc0ef85d0dd2a already in progress: creating tablet
I20250109 20:52:43.028033 1160 log.cc:1198] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/920a6be9db344c098c62bfd2fa5a73e8
I20250109 20:52:43.028318 1160 ts_tablet_manager.cc:1939] T 920a6be9db344c098c62bfd2fa5a73e8 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:43.029366 893 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.031077 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:43.031116 1157 ts_tablet_manager.cc:1905] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.032119 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:43.032888 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cc174d6bd9f44eaf9f66e54693597601 because the tablet was not found. No further retry: Not found: Tablet not found: cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:43.033726 893 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.035404 655 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:43.035818 753 tablet_service.cc:1514] Processing DeleteTablet for tablet bd445a1adba34529aa3f7a44b991b177 with delete_type TABLET_DATA_DELETED (replaced by tablet 5496b2f965064f7788ebd74c62d1240a) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.037154 890 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.037494 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 920a6be9db344c098c62bfd2fa5a73e8 (table default.Table_eb403509cf414f548af84d837a6f04ac [id=a1dea7259a764d7ca62d5ff76ccc95b8]) successfully deleted
I20250109 20:52:43.037904 754 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.038137 1157 ts_tablet_manager.cc:1918] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.038259 1157 log.cc:1198] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/899f08250f2248778f6abab551093461
I20250109 20:52:43.038281 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.038738 1176 tablet_bootstrap.cc:492] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.038895 1176 ts_tablet_manager.cc:1397] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.095s user 0.014s sys 0.023s
W20250109 20:52:43.039178 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cc174d6bd9f44eaf9f66e54693597601 because the tablet was not found. No further retry: Not found: Tablet not found: cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:43.039620 1176 raft_consensus.cc:357] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.039803 1176 raft_consensus.cc:383] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.039898 1176 raft_consensus.cc:738] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.040138 1176 consensus_queue.cc:260] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.041388 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9af6ca0c3f0042358b13f0891ced6dea" } modify_external_catalogs: true
I20250109 20:52:43.041671 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9af6ca0c3f0042358b13f0891ced6dea" } modify_external_catalogs: true
I20250109 20:52:43.042028 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7ed9e8c921491c9d911d662f500451 (DEFAULT_TABLE table=default.Table_b2dd9dd1ca5c4c1b9d2925b5110c81e5 [id=53718e55e4344456851f408909cc8170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.042354 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7ed9e8c921491c9d911d662f500451. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.046010 752 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 (DEFAULT_TABLE table=default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.046384 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5a90e65e80494a8b451f2bc6baa2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.046700 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 (DEFAULT_TABLE table=default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.048393 751 tablet_service.cc:1467] Processing CreateTablet for tablet 175bf6c335a8405c97b296cf4718610b (DEFAULT_TABLE table=default.Table_f580256f4281489b85b9f46fcd381c5f [id=a6f0a5fcb43d4509a188259adfdd0192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.048905 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 175bf6c335a8405c97b296cf4718610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.050100 1176 ts_tablet_manager.cc:1428] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250109 20:52:43.050318 1176 tablet_bootstrap.cc:492] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.053704 1176 tablet_bootstrap.cc:654] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.056280 754 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7ed9e8c921491c9d911d662f500451 (DEFAULT_TABLE table=default.Table_b2dd9dd1ca5c4c1b9d2925b5110c81e5 [id=53718e55e4344456851f408909cc8170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.056600 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7ed9e8c921491c9d911d662f500451. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.057123 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]). Expected version 1 got 0
I20250109 20:52:43.061231 1176 tablet_bootstrap.cc:492] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.061376 1176 ts_tablet_manager.cc:1397] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20250109 20:52:43.062089 1176 raft_consensus.cc:357] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.062274 1176 raft_consensus.cc:383] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.062358 1176 raft_consensus.cc:738] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.062613 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8c0e3ac68686429dadf4fcc158cf285b" } modify_external_catalogs: true
I20250109 20:52:43.062709 1176 consensus_queue.cc:260] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.062968 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8c0e3ac68686429dadf4fcc158cf285b" } modify_external_catalogs: true
I20250109 20:52:43.064934 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5496b2f965064f7788ebd74c62d1240a
I20250109 20:52:43.064936 1176 ts_tablet_manager.cc:1428] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:43.065039 1157 ts_tablet_manager.cc:1939] T 899f08250f2248778f6abab551093461 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:43.065460 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5a90e65e80494a8b451f2bc6baa2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.065907 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 899f08250f2248778f6abab551093461 (table default.Table_732fb1f4c69449e8950e5fa85c7a2948 [id=a667d9b2080d44b89a1715b9b57c8e8b]) successfully deleted
W20250109 20:52:43.067214 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c6a3dbc983534d65994fc0ef85d0dd2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a3dbc983534d65994fc0ef85d0dd2a already in progress: creating tablet
I20250109 20:52:43.067600 1176 tablet_bootstrap.cc:492] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:43.068127 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cc174d6bd9f44eaf9f66e54693597601 because the tablet was not found. No further retry: Not found: Tablet not found: cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:43.068431 753 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.068902 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 (DEFAULT_TABLE table=default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.069262 1176 tablet_bootstrap.cc:654] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:43.069676 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cc174d6bd9f44eaf9f66e54693597601 because the tablet was not found. No further retry: Not found: Tablet not found: cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:43.069262 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5a90e65e80494a8b451f2bc6baa2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.070459 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a58d6a9aabde40128ad0585f772ef591" } modify_external_catalogs: true
I20250109 20:52:43.070611 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a58d6a9aabde40128ad0585f772ef591" } modify_external_catalogs: true
I20250109 20:52:43.072849 1160 tablet_replica.cc:331] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:43.073031 1160 raft_consensus.cc:2238] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.073382 1160 raft_consensus.cc:2267] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.076818 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]). Expected version 1 got 0
I20250109 20:52:43.076838 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7ed9e8c921491c9d911d662f500451 (DEFAULT_TABLE table=default.Table_b2dd9dd1ca5c4c1b9d2925b5110c81e5 [id=53718e55e4344456851f408909cc8170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.077159 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7ed9e8c921491c9d911d662f500451. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.077179 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b133f574cb3d4a08b0318b2c3189b540
I20250109 20:52:43.078559 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d2aacfc7eea3490dbeb9d5c82d48766d
I20250109 20:52:43.079466 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:43.080053 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d2aacfc7eea3490dbeb9d5c82d48766d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2aacfc7eea3490dbeb9d5c82d48766d already in progress: creating tablet
I20250109 20:52:43.080319 753 tablet_service.cc:1514] Processing DeleteTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:43.080817 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d2aacfc7eea3490dbeb9d5c82d48766d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2aacfc7eea3490dbeb9d5c82d48766d already in progress: creating tablet
I20250109 20:52:43.081149 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 175bf6c335a8405c97b296cf4718610b (DEFAULT_TABLE table=default.Table_f580256f4281489b85b9f46fcd381c5f [id=a6f0a5fcb43d4509a188259adfdd0192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.081475 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 175bf6c335a8405c97b296cf4718610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.081861 895 tablet_service.cc:1514] Processing DeleteTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.082355 899 tablet_service.cc:1467] Processing CreateTablet for tablet 175bf6c335a8405c97b296cf4718610b (DEFAULT_TABLE table=default.Table_f580256f4281489b85b9f46fcd381c5f [id=a6f0a5fcb43d4509a188259adfdd0192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.082655 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 175bf6c335a8405c97b296cf4718610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.085377 888 tablet_service.cc:1467] Processing CreateTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 (DEFAULT_TABLE table=default.Table_49d1907147ef4990876347467005e31e [id=f06f85dff8684b1cac234aa2e35b6598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.085435 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ae6c4e835048bd8deed5ac9c4ae07a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.085668 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e06e64448a4fb2bf41fc6e3d0361a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.085896 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 97205d09de264adb9d19cb32546b0942 (DEFAULT_TABLE table=default.Table_1bca442744ee4e04a46973fb21434cae [id=45220eb0588a4c62838a951101e7e04f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.086203 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97205d09de264adb9d19cb32546b0942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.086148 895 tablet_service.cc:1467] Processing CreateTablet for tablet 97205d09de264adb9d19cb32546b0942 (DEFAULT_TABLE table=default.Table_1bca442744ee4e04a46973fb21434cae [id=45220eb0588a4c62838a951101e7e04f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.086416 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97205d09de264adb9d19cb32546b0942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.087466 752 tablet_service.cc:1467] Processing CreateTablet for tablet 47f2f62aab954531ad6eb60b6f4661b8 (DEFAULT_TABLE table=default.Table_e2cedbf0f0cd4d88ae098ddd22cb9972 [id=69b26a1bf8564a889d1227cfad55d0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.087878 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f2f62aab954531ad6eb60b6f4661b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.088943 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 (DEFAULT_TABLE table=default.Table_49d1907147ef4990876347467005e31e [id=f06f85dff8684b1cac234aa2e35b6598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.089272 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e06e64448a4fb2bf41fc6e3d0361a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.089344 1160 ts_tablet_manager.cc:1905] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.093529 1166 tablet_replica.cc:331] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:43.093752 1166 raft_consensus.cc:2238] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.093921 1166 raft_consensus.cc:2267] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.096742 898 tablet_service.cc:1467] Processing CreateTablet for tablet 47f2f62aab954531ad6eb60b6f4661b8 (DEFAULT_TABLE table=default.Table_e2cedbf0f0cd4d88ae098ddd22cb9972 [id=69b26a1bf8564a889d1227cfad55d0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.097342 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f2f62aab954531ad6eb60b6f4661b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.097779 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 47f2f62aab954531ad6eb60b6f4661b8 (DEFAULT_TABLE table=default.Table_e2cedbf0f0cd4d88ae098ddd22cb9972 [id=69b26a1bf8564a889d1227cfad55d0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.098120 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f2f62aab954531ad6eb60b6f4661b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.099395 757 tablet_service.cc:1467] Processing CreateTablet for tablet cc174d6bd9f44eaf9f66e54693597601 (DEFAULT_TABLE table=default.Table_c576a7a39c174fe6ac5c351e1b4f80d3 [id=37a12f94e9c54332af6b774b56b13666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.099730 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc174d6bd9f44eaf9f66e54693597601. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.100430 1034 tablet_service.cc:1467] Processing CreateTablet for tablet cc174d6bd9f44eaf9f66e54693597601 (DEFAULT_TABLE table=default.Table_c576a7a39c174fe6ac5c351e1b4f80d3 [id=37a12f94e9c54332af6b774b56b13666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.100729 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc174d6bd9f44eaf9f66e54693597601. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.103325 753 tablet_service.cc:1467] Processing CreateTablet for tablet 97205d09de264adb9d19cb32546b0942 (DEFAULT_TABLE table=default.Table_1bca442744ee4e04a46973fb21434cae [id=45220eb0588a4c62838a951101e7e04f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.103698 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97205d09de264adb9d19cb32546b0942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.105919 896 tablet_service.cc:1467] Processing CreateTablet for tablet cc174d6bd9f44eaf9f66e54693597601 (DEFAULT_TABLE table=default.Table_c576a7a39c174fe6ac5c351e1b4f80d3 [id=37a12f94e9c54332af6b774b56b13666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.106235 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc174d6bd9f44eaf9f66e54693597601. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.107685 1166 ts_tablet_manager.cc:1905] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.113837 1166 ts_tablet_manager.cc:1918] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.113988 1166 log.cc:1198] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/30ae6c4e835048bd8deed5ac9c4ae07a
I20250109 20:52:43.114423 1166 ts_tablet_manager.cc:1939] T 30ae6c4e835048bd8deed5ac9c4ae07a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:43.115378 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 30ae6c4e835048bd8deed5ac9c4ae07a (table default.Table_a6d4d40a237548c384fd897eb207903c [id=6fe0e6310b9f44a8b3b9558fec22c2a6]) successfully deleted
I20250109 20:52:43.115924 751 tablet_service.cc:1467] Processing CreateTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 (DEFAULT_TABLE table=default.Table_49d1907147ef4990876347467005e31e [id=f06f85dff8684b1cac234aa2e35b6598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.116302 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e06e64448a4fb2bf41fc6e3d0361a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.131438 1160 ts_tablet_manager.cc:1918] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.132861 1160 log.cc:1198] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ecee7292068c4d649a1232ac54dbb7f0
I20250109 20:52:43.133368 1160 ts_tablet_manager.cc:1939] T ecee7292068c4d649a1232ac54dbb7f0 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:43.133492 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7a97ba56daf04341bc72ea1ffe462b8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.134253 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a97ba56daf04341bc72ea1ffe462b8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.134383 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ecee7292068c4d649a1232ac54dbb7f0 (table default.Table_7a8859a022a54a48affdc0ef3024c194 [id=315670c6ced1467e82fc224f50ca5625]) successfully deleted
I20250109 20:52:43.135565 1160 tablet_replica.cc:331] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:43.135742 1160 raft_consensus.cc:2238] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.135867 1160 raft_consensus.cc:2267] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.135797 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5285d784ef3043f397f8b3f5c0facc07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.136289 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5285d784ef3043f397f8b3f5c0facc07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.136682 1160 ts_tablet_manager.cc:1905] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.142557 1160 ts_tablet_manager.cc:1918] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.142685 1160 log.cc:1198] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/899f08250f2248778f6abab551093461
I20250109 20:52:43.142833 1176 tablet_bootstrap.cc:492] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.142962 1176 ts_tablet_manager.cc:1397] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20250109 20:52:43.143029 1160 ts_tablet_manager.cc:1939] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:43.143296 449 master_service.cc:946] ReplaceTablet: received request to replace tablet e5d6dc79bf1e48ee9cf2bbb70463741d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.143728 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 899f08250f2248778f6abab551093461 (table default.Table_732fb1f4c69449e8950e5fa85c7a2948 [id=a667d9b2080d44b89a1715b9b57c8e8b]) successfully deleted
W20250109 20:52:43.144243 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c6a3dbc983534d65994fc0ef85d0dd2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a3dbc983534d65994fc0ef85d0dd2a already in progress: creating tablet
W20250109 20:52:43.144474 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cc174d6bd9f44eaf9f66e54693597601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc174d6bd9f44eaf9f66e54693597601 already in progress: creating tablet
W20250109 20:52:43.144646 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cc174d6bd9f44eaf9f66e54693597601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc174d6bd9f44eaf9f66e54693597601 already in progress: creating tablet
W20250109 20:52:43.144836 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d2aacfc7eea3490dbeb9d5c82d48766d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2aacfc7eea3490dbeb9d5c82d48766d already in progress: creating tablet
I20250109 20:52:43.145385 1158 tablet_bootstrap.cc:492] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.145511 1158 ts_tablet_manager.cc:1397] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.126s user 0.019s sys 0.047s
I20250109 20:52:43.146301 1158 raft_consensus.cc:357] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.146477 1158 raft_consensus.cc:383] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.146562 1158 raft_consensus.cc:738] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.146775 1158 consensus_queue.cc:260] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.147253 1158 ts_tablet_manager.cc:1428] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:43.147487 1158 tablet_bootstrap.cc:492] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.147859 1176 raft_consensus.cc:357] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.149142 1158 tablet_bootstrap.cc:654] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.149399 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c9d4e1c296d042c0be38c03d21165799"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.149901 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9d4e1c296d042c0be38c03d21165799 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.152136 1176 raft_consensus.cc:383] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.152235 1176 raft_consensus.cc:738] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.152474 1176 consensus_queue.cc:260] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.154847 454 master_service.cc:946] ReplaceTablet: received request to replace tablet e5d6dc79bf1e48ee9cf2bbb70463741d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.156698 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_82b978780ede49bfaa599c5af6e7d57d [id=e4a51641b240493d9f69b9031ef5d4b1]). Expected version 2 got 0
I20250109 20:52:43.159466 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e5d6dc79bf1e48ee9cf2bbb70463741d
I20250109 20:52:43.159969 449 catalog_manager.cc:6596] ReplaceTablet: tablet e5d6dc79bf1e48ee9cf2bbb70463741d deleted and replaced by tablet 66c6016d88b54eb390f880945ddb1ea6
I20250109 20:52:43.160492 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d with delete_type TABLET_DATA_DELETED (replaced by tablet 66c6016d88b54eb390f880945ddb1ea6) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.160835 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d with delete_type TABLET_DATA_DELETED (replaced by tablet 66c6016d88b54eb390f880945ddb1ea6) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.161163 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d with delete_type TABLET_DATA_DELETED (replaced by tablet 66c6016d88b54eb390f880945ddb1ea6) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:43.161669 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e5d6dc79bf1e48ee9cf2bbb70463741d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5d6dc79bf1e48ee9cf2bbb70463741d already in progress: creating tablet
W20250109 20:52:43.161772 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e5d6dc79bf1e48ee9cf2bbb70463741d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5d6dc79bf1e48ee9cf2bbb70463741d already in progress: creating tablet
W20250109 20:52:43.162045 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e5d6dc79bf1e48ee9cf2bbb70463741d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5d6dc79bf1e48ee9cf2bbb70463741d already in progress: creating tablet
I20250109 20:52:43.164482 1176 ts_tablet_manager.cc:1428] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250109 20:52:43.164832 1176 tablet_bootstrap.cc:492] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.168653 1176 tablet_bootstrap.cc:654] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.169412 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.169657 1166 tablet_replica.cc:331] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:43.169821 1166 raft_consensus.cc:2238] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.169979 1166 raft_consensus.cc:2267] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.170867 1166 ts_tablet_manager.cc:1905] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:43.178207 854 leader_election.cc:343] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 899f08250f2248778f6abab551093461
W20250109 20:52:43.182138 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 58817dddbe8749249a96c60dee41b520 (table default.Table_68ea6c2a7f6e4abb9728c2ab20e3b17e [id=fc46d1a61d7d4769bfcf249712d2e2b9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 58817dddbe8749249a96c60dee41b520 consensus state has no leader
I20250109 20:52:43.185137 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b2dd9dd1ca5c4c1b9d2925b5110c81e5" } new_table_name: "default.Table_78db655e42e74268bbdff89f440ddc18" modify_external_catalogs: true
I20250109 20:52:43.188455 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_08e4b27362104ad4af8816b4facabed7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.189071 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08e4b27362104ad4af8816b4facabed7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.189193 447 master_service.cc:946] ReplaceTablet: received request to replace tablet bd445a1adba34529aa3f7a44b991b177 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.189155 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bb350b3552b34845ab84e306036fe133"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.189586 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb350b3552b34845ab84e306036fe133 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.190331 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_51da7d20722144bf824bdc0388d68808"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.190717 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51da7d20722144bf824bdc0388d68808 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.191717 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0504385486ba4dd18a9a56a3f206aa9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.192108 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0504385486ba4dd18a9a56a3f206aa9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.192281 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8e9a23e5c7864ef78f06146e2469dc3c" } new_table_name: "default.Table_5ab3d2823bee4f739f414db53a1a6cd9" modify_external_catalogs: true
I20250109 20:52:43.193295 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_82b8503333e64f94829491f22a40b92b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.193699 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82b8503333e64f94829491f22a40b92b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.194489 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_63b84b4a77ba41198819de24f972ae17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.194793 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63b84b4a77ba41198819de24f972ae17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.194918 1166 ts_tablet_manager.cc:1918] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.195062 1166 log.cc:1198] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6c38c8f0299e49ff8aff99b17b7be01b
I20250109 20:52:43.195484 1166 ts_tablet_manager.cc:1939] T 6c38c8f0299e49ff8aff99b17b7be01b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:43.195449 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_807cf97a78f84ba794b3d29b6e210081"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.195735 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_807cf97a78f84ba794b3d29b6e210081 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.196705 1165 tablet_bootstrap.cc:492] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:43.196911 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6c38c8f0299e49ff8aff99b17b7be01b (table default.Table_70760925d6df4f0c961c825337270e1a [id=d9fb05f1ceef49638bbe6db78d2d17c2]) successfully deleted
I20250109 20:52:43.196941 1165 ts_tablet_manager.cc:1397] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.254s user 0.001s sys 0.003s
I20250109 20:52:43.197607 1165 raft_consensus.cc:357] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.198521 1165 raft_consensus.cc:383] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.198942 1165 raft_consensus.cc:738] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.199447 1165 consensus_queue.cc:260] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.202387 1158 tablet_bootstrap.cc:492] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.202551 1158 ts_tablet_manager.cc:1397] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.024s
I20250109 20:52:43.203289 1158 raft_consensus.cc:357] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.203507 1158 raft_consensus.cc:383] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.203586 1158 raft_consensus.cc:738] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.203840 1158 consensus_queue.cc:260] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.204499 1158 ts_tablet_manager.cc:1428] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:43.204850 1158 tablet_bootstrap.cc:492] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.205758 751 tablet_service.cc:1467] Processing CreateTablet for tablet 268d9969a46544d9954f71c77170e76c (DEFAULT_TABLE table=default.Table_5285d784ef3043f397f8b3f5c0facc07 [id=42597c8e893d4f7084df9638e72d0306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.206094 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268d9969a46544d9954f71c77170e76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.207491 1158 tablet_bootstrap.cc:654] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.208037 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 175bf6c335a8405c97b296cf4718610b from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.208807 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_093ba0191f3b46f0aa7b6eb7c40f6f69" } new_table_name: "default.Table_71377d92f20d4047a2af17e649b9ac62" modify_external_catalogs: true
I20250109 20:52:43.209690 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 268d9969a46544d9954f71c77170e76c (DEFAULT_TABLE table=default.Table_5285d784ef3043f397f8b3f5c0facc07 [id=42597c8e893d4f7084df9638e72d0306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.210150 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268d9969a46544d9954f71c77170e76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.210564 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_68ea6c2a7f6e4abb9728c2ab20e3b17e" } new_table_name: "default.Table_d0a7a37106ea46e683ead0b81aa0585e" modify_external_catalogs: true
W20250109 20:52:43.212878 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bea28fb446e49669133e21b02c416ad consensus state has no leader
W20250109 20:52:43.216338 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c7ed9e8c921491c9d911d662f500451 consensus state has no leader
I20250109 20:52:43.218829 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 75ec6b6b1ec1462d82008a78a885032d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.219079 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e440d446aaba4d62a8ea1ad840f9147c" } new_table_name: "default.Table_1bb496e68f2e45c39a6a4adcc8db8526" modify_external_catalogs: true
I20250109 20:52:43.230953 888 tablet_service.cc:1467] Processing CreateTablet for tablet 268d9969a46544d9954f71c77170e76c (DEFAULT_TABLE table=default.Table_5285d784ef3043f397f8b3f5c0facc07 [id=42597c8e893d4f7084df9638e72d0306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.231483 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268d9969a46544d9954f71c77170e76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.231622 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5c35bca9b68e42bb9075f51687f74d15" } modify_external_catalogs: true
I20250109 20:52:43.231815 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5c35bca9b68e42bb9075f51687f74d15" } modify_external_catalogs: true
I20250109 20:52:43.238839 1165 ts_tablet_manager.cc:1428] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.042s user 0.001s sys 0.000s
I20250109 20:52:43.239230 1165 tablet_bootstrap.cc:492] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:43.241149 1165 tablet_bootstrap.cc:654] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.245666 1165 tablet_bootstrap.cc:492] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:43.245826 1165 ts_tablet_manager.cc:1397] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250109 20:52:43.245914 1176 tablet_bootstrap.cc:492] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.246065 1176 ts_tablet_manager.cc:1397] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.081s user 0.012s sys 0.019s
I20250109 20:52:43.246743 1165 raft_consensus.cc:357] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.246965 1165 raft_consensus.cc:383] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.247040 1165 raft_consensus.cc:738] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.247263 1165 consensus_queue.cc:260] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.247669 1176 raft_consensus.cc:357] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.247851 1176 raft_consensus.cc:383] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.247911 1176 raft_consensus.cc:738] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.248189 1165 ts_tablet_manager.cc:1428] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:43.248142 1176 consensus_queue.cc:260] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.248454 1165 tablet_bootstrap.cc:492] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:43.250219 1176 ts_tablet_manager.cc:1428] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:43.250543 1176 tablet_bootstrap.cc:492] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.250600 1165 tablet_bootstrap.cc:654] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.252254 1176 tablet_bootstrap.cc:654] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.252422 1158 tablet_bootstrap.cc:492] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.252552 1158 ts_tablet_manager.cc:1397] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.048s user 0.021s sys 0.004s
I20250109 20:52:43.253316 1158 raft_consensus.cc:357] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.253513 1158 raft_consensus.cc:383] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.253582 1158 raft_consensus.cc:738] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.253777 1158 consensus_queue.cc:260] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.254879 1158 ts_tablet_manager.cc:1428] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:43.255112 1158 tablet_bootstrap.cc:492] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.255194 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7a97ba56daf04341bc72ea1ffe462b8c" } modify_external_catalogs: true
I20250109 20:52:43.255381 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7a97ba56daf04341bc72ea1ffe462b8c" } modify_external_catalogs: true
I20250109 20:52:43.255712 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 920a6be9db344c098c62bfd2fa5a73e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:43.255781 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58817dddbe8749249a96c60dee41b520 consensus state has no leader
I20250109 20:52:43.255993 1166 tablet_replica.cc:331] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:43.256171 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ded8e3a447c84b9b8ec4ccd70199ccd5 consensus state has no leader
I20250109 20:52:43.256209 1166 raft_consensus.cc:2238] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.256356 1166 raft_consensus.cc:2267] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.256764 1158 tablet_bootstrap.cc:654] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.256973 1166 ts_tablet_manager.cc:1905] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:43.257947 854 leader_election.cc:343] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 899f08250f2248778f6abab551093461
I20250109 20:52:43.258086 854 leader_election.cc:304] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:52:43.258388 854 raft_consensus.cc:2681] T 899f08250f2248778f6abab551093461 P 44d473955919435cb679edd5f8afe2c2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:52:43.264703 1176 tablet_bootstrap.cc:492] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.264868 1176 ts_tablet_manager.cc:1397] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20250109 20:52:43.265749 1176 raft_consensus.cc:357] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.265949 1176 raft_consensus.cc:383] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.266070 1176 raft_consensus.cc:738] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.266281 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49a9931b8903487db0856c10d60fc1e9" } modify_external_catalogs: true
I20250109 20:52:43.266420 1166 ts_tablet_manager.cc:1918] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.266310 1176 consensus_queue.cc:260] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.266427 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49a9931b8903487db0856c10d60fc1e9" } modify_external_catalogs: true
I20250109 20:52:43.266531 1166 log.cc:1198] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/920a6be9db344c098c62bfd2fa5a73e8
I20250109 20:52:43.266884 1166 ts_tablet_manager.cc:1939] T 920a6be9db344c098c62bfd2fa5a73e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:43.267793 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 920a6be9db344c098c62bfd2fa5a73e8 (table default.Table_eb403509cf414f548af84d837a6f04ac [id=a1dea7259a764d7ca62d5ff76ccc95b8]) successfully deleted
I20250109 20:52:43.267879 1176 ts_tablet_manager.cc:1428] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:43.268133 1176 tablet_bootstrap.cc:492] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.268988 1158 tablet_bootstrap.cc:492] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.269143 1158 ts_tablet_manager.cc:1397] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20250109 20:52:43.269531 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ec6b6b1ec1462d82008a78a885032d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.269780 1166 tablet_replica.cc:331] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:43.269924 1166 raft_consensus.cc:2238] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.269889 1158 raft_consensus.cc:357] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.270067 1158 raft_consensus.cc:383] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.270073 1166 raft_consensus.cc:2267] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.270129 1158 raft_consensus.cc:738] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.270334 1158 consensus_queue.cc:260] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.270761 1166 ts_tablet_manager.cc:1905] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.270853 1158 ts_tablet_manager.cc:1428] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:43.271106 1158 tablet_bootstrap.cc:492] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.273222 1158 tablet_bootstrap.cc:654] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:43.278178 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab593a602d61495eb7b323903ae610b6 consensus state has no leader
I20250109 20:52:43.279147 1166 ts_tablet_manager.cc:1918] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.279244 1166 log.cc:1198] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/75ec6b6b1ec1462d82008a78a885032d
I20250109 20:52:43.279542 1166 ts_tablet_manager.cc:1939] T 75ec6b6b1ec1462d82008a78a885032d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:43.280650 446 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet cdd067c335354357b44dac433f8c1523 (table default.Table_fd20b680db98408db7e7f0278832db97 [id=8c860f8c96254692906f892a25530962]). Expected version 3 got 0
I20250109 20:52:43.281244 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 75ec6b6b1ec1462d82008a78a885032d (table default.Table_1d87425867fc462e86d1f51d3c7777dd [id=289e52afdcea437bb60cd657c8dc01d1]) successfully deleted
I20250109 20:52:43.281384 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_82b978780ede49bfaa599c5af6e7d57d [id=e4a51641b240493d9f69b9031ef5d4b1]). Expected version 2 got 0
I20250109 20:52:43.281746 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bb350b3552b34845ab84e306036fe133" } modify_external_catalogs: true
I20250109 20:52:43.281955 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bb350b3552b34845ab84e306036fe133" } modify_external_catalogs: true
I20250109 20:52:43.284054 1176 tablet_bootstrap.cc:654] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.285382 1187 raft_consensus.cc:491] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:43.285626 1187 raft_consensus.cc:513] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.286815 888 tablet_service.cc:1467] Processing CreateTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 (DEFAULT_TABLE table=default.Table_c9d4e1c296d042c0be38c03d21165799 [id=756ad36069be45acaadb834d6ad41d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.287240 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32eb4dfcf96c4a05a55082e90ae04467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.287741 1187 leader_election.cc:290] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:43.287933 1158 tablet_bootstrap.cc:492] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.288125 1158 ts_tablet_manager.cc:1397] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20250109 20:52:43.288911 1158 raft_consensus.cc:357] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.289099 1158 raft_consensus.cc:383] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.289166 1158 raft_consensus.cc:738] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.289402 1158 consensus_queue.cc:260] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.291767 1158 ts_tablet_manager.cc:1428] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:52:43.293877 1158 tablet_bootstrap.cc:492] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.294219 751 tablet_service.cc:1467] Processing CreateTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 (DEFAULT_TABLE table=default.Table_c9d4e1c296d042c0be38c03d21165799 [id=756ad36069be45acaadb834d6ad41d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.294667 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32eb4dfcf96c4a05a55082e90ae04467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.296010 1158 tablet_bootstrap.cc:654] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.302181 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d0a7a37106ea46e683ead0b81aa0585e" } modify_external_catalogs: true
I20250109 20:52:43.303473 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet cdd067c335354357b44dac433f8c1523 (table default.Table_fd20b680db98408db7e7f0278832db97 [id=8c860f8c96254692906f892a25530962]). Expected version 3 got 0
I20250109 20:52:43.307663 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d0a7a37106ea46e683ead0b81aa0585e" } modify_external_catalogs: true
I20250109 20:52:43.310669 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 (DEFAULT_TABLE table=default.Table_c9d4e1c296d042c0be38c03d21165799 [id=756ad36069be45acaadb834d6ad41d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.310791 753 tablet_service.cc:1467] Processing CreateTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 (DEFAULT_TABLE table=default.Table_7a97ba56daf04341bc72ea1ffe462b8c [id=9c14de13a61b45f78bacd804507a2b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.311156 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32eb4dfcf96c4a05a55082e90ae04467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.311245 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825efab5ff7b492ca3c7e3fe11ece098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.315968 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ab00e387fbc843b2841bef2d3289df34
I20250109 20:52:43.316031 754 tablet_service.cc:1467] Processing CreateTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 (DEFAULT_TABLE table=default.Table_f9f28decdfb64455809f7b283552a4a5 [id=ae7fb9326c2c49a593d1111d45ea424f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.316360 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c6016d88b54eb390f880945ddb1ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.317092 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 34cfa3005fdd40709a630d5bb80564db
I20250109 20:52:43.320200 757 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee7292068c4d649a1232ac54dbb7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.320448 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454b27918ab41ad9ab269185b2692fe with delete_type TABLET_DATA_DELETED (replaced by tablet 35d7cead4f4149ae8de48eee2cfd8679) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.320478 1157 tablet_replica.cc:331] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:43.320693 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ec6b6b1ec1462d82008a78a885032d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.321246 1157 raft_consensus.cc:2238] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.321426 1157 raft_consensus.cc:2267] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.322213 1157 ts_tablet_manager.cc:1905] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.322328 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 825efab5ff7b492ca3c7e3fe11ece098
I20250109 20:52:43.323127 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 39da4ebe91ff425998b5a8f78aa008c4
I20250109 20:52:43.327316 446 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cdd067c335354357b44dac433f8c1523 (table default.Table_fd20b680db98408db7e7f0278832db97 [id=8c860f8c96254692906f892a25530962]). Expected version 3 got 0
W20250109 20:52:43.329568 627 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:56868 (request call id 23) took 828 ms (client timeout 1000 ms). Trace:
W20250109 20:52:43.329749 627 rpcz_store.cc:269] 0109 20:52:42.501503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:42.501562 (+ 59us) service_pool.cc:225] Handling call
0109 20:52:43.329476 (+827914us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250109 20:52:43.334182 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu with pid 564
I20250109 20:52:43.335690 1157 ts_tablet_manager.cc:1918] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.335824 1157 log.cc:1198] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ecee7292068c4d649a1232ac54dbb7f0
I20250109 20:52:43.336272 1157 ts_tablet_manager.cc:1939] T ecee7292068c4d649a1232ac54dbb7f0 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:43.337373 1157 tablet_replica.cc:331] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:43.337623 1157 raft_consensus.cc:2238] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.337805 1157 raft_consensus.cc:2267] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.337834 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cfa3005fdd40709a630d5bb80564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.338128 755 tablet_service.cc:1514] Processing DeleteTablet for tablet ab00e387fbc843b2841bef2d3289df34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.338914 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 175bf6c335a8405c97b296cf4718610b
I20250109 20:52:43.339433 449 catalog_manager.cc:6596] ReplaceTablet: tablet 175bf6c335a8405c97b296cf4718610b deleted and replaced by tablet 11d303cd42a2428fa17f29b48c4f64f5
I20250109 20:52:43.339911 1157 ts_tablet_manager.cc:1905] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.340430 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ecee7292068c4d649a1232ac54dbb7f0 (table default.Table_7a8859a022a54a48affdc0ef3024c194 [id=315670c6ced1467e82fc224f50ca5625]) successfully deleted
I20250109 20:52:43.341192 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 175bf6c335a8405c97b296cf4718610b with delete_type TABLET_DATA_DELETED (replaced by tablet 11d303cd42a2428fa17f29b48c4f64f5) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.341483 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.341943 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 58817dddbe8749249a96c60dee41b520
I20250109 20:52:43.343817 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cfa3005fdd40709a630d5bb80564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.344221 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:43.344974 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 34cfa3005fdd40709a630d5bb80564db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34cfa3005fdd40709a630d5bb80564db already in progress: creating tablet
I20250109 20:52:43.344992 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 (DEFAULT_TABLE table=default.Table_f9f28decdfb64455809f7b283552a4a5 [id=ae7fb9326c2c49a593d1111d45ea424f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:43.345253 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 825efab5ff7b492ca3c7e3fe11ece098 because the tablet was not found. No further retry: Not found: Tablet not found: 825efab5ff7b492ca3c7e3fe11ece098
I20250109 20:52:43.345449 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c6016d88b54eb390f880945ddb1ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.347024 1165 tablet_bootstrap.cc:492] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:43.347195 1165 ts_tablet_manager.cc:1397] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.099s user 0.007s sys 0.006s
I20250109 20:52:43.348125 1165 raft_consensus.cc:357] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.348353 1165 raft_consensus.cc:383] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.348433 1165 raft_consensus.cc:738] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.348958 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ab00e387fbc843b2841bef2d3289df34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.349844 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 58817dddbe8749249a96c60dee41b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.350006 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 175bf6c335a8405c97b296cf4718610b with delete_type TABLET_DATA_DELETED (replaced by tablet 11d303cd42a2428fa17f29b48c4f64f5) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:43.350816 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ab00e387fbc843b2841bef2d3289df34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab00e387fbc843b2841bef2d3289df34 already in progress: creating tablet
W20250109 20:52:43.351035 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 58817dddbe8749249a96c60dee41b520 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58817dddbe8749249a96c60dee41b520 already in progress: creating tablet
W20250109 20:52:43.351262 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 175bf6c335a8405c97b296cf4718610b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 175bf6c335a8405c97b296cf4718610b already in progress: creating tablet
I20250109 20:52:43.351460 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 (DEFAULT_TABLE table=default.Table_7a97ba56daf04341bc72ea1ffe462b8c [id=9c14de13a61b45f78bacd804507a2b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.351802 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825efab5ff7b492ca3c7e3fe11ece098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.352021 896 tablet_service.cc:1467] Processing CreateTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 (DEFAULT_TABLE table=default.Table_f9f28decdfb64455809f7b283552a4a5 [id=ae7fb9326c2c49a593d1111d45ea424f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.352365 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c6016d88b54eb390f880945ddb1ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.360036 888 tablet_service.cc:1467] Processing CreateTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 (DEFAULT_TABLE table=default.Table_7a97ba56daf04341bc72ea1ffe462b8c [id=9c14de13a61b45f78bacd804507a2b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.360432 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825efab5ff7b492ca3c7e3fe11ece098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.361195 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.361898 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.362268 1157 ts_tablet_manager.cc:1918] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.362401 1157 log.cc:1198] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/0454b27918ab41ad9ab269185b2692fe
I20250109 20:52:43.362782 1157 ts_tablet_manager.cc:1939] T 0454b27918ab41ad9ab269185b2692fe P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:43.363448 895 tablet_service.cc:1514] Processing DeleteTablet for tablet ab00e387fbc843b2841bef2d3289df34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.363760 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 0454b27918ab41ad9ab269185b2692fe (table default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
I20250109 20:52:43.363978 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cfa3005fdd40709a630d5bb80564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.365417 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8389e66e581f406bb82735a1ece78773 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.365702 1160 tablet_replica.cc:331] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:43.365909 1160 raft_consensus.cc:2238] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.366111 1160 raft_consensus.cc:2267] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.366842 1160 ts_tablet_manager.cc:1905] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.376142 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 58817dddbe8749249a96c60dee41b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.378595 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9214f432e5c04dca831c8a093b699430"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:43.378955 1160 ts_tablet_manager.cc:1918] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.379079 1160 log.cc:1198] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/8389e66e581f406bb82735a1ece78773
W20250109 20:52:43.379251 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9214f432e5c04dca831c8a093b699430 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.379472 1160 ts_tablet_manager.cc:1939] T 8389e66e581f406bb82735a1ece78773 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:43.380824 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.391657 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 75ec6b6b1ec1462d82008a78a885032d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.391976 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 89fb6aeedc77452f99c46f03d6d8d089 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.392853 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 175bf6c335a8405c97b296cf4718610b with delete_type TABLET_DATA_DELETED (replaced by tablet 11d303cd42a2428fa17f29b48c4f64f5) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.393369 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 58817dddbe8749249a96c60dee41b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:43.394068 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ab00e387fbc843b2841bef2d3289df34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab00e387fbc843b2841bef2d3289df34 already in progress: creating tablet
W20250109 20:52:43.394531 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 34cfa3005fdd40709a630d5bb80564db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34cfa3005fdd40709a630d5bb80564db already in progress: creating tablet
I20250109 20:52:43.394932 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 8389e66e581f406bb82735a1ece78773 (table default.Table_39dd6100bb4044f6a89213a6875f8756 [id=d641ab17c06741d4a0816205d1f4b16a]) successfully deleted
W20250109 20:52:43.395403 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 825efab5ff7b492ca3c7e3fe11ece098 because the tablet was not found. No further retry: Not found: Tablet not found: 825efab5ff7b492ca3c7e3fe11ece098
I20250109 20:52:43.397981 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 89fb6aeedc77452f99c46f03d6d8d089 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.398247 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_edf77c78245e46578d5b29f18ab58b40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.398906 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edf77c78245e46578d5b29f18ab58b40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.399632 1157 tablet_replica.cc:331] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:43.400552 1157 raft_consensus.cc:2238] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.400804 1157 raft_consensus.cc:2267] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.401772 1157 ts_tablet_manager.cc:1905] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.406841 1178 raft_consensus.cc:491] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:43.407011 1178 raft_consensus.cc:513] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.407632 1178 leader_election.cc:290] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:43.408458 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38760b86fdfb4c8c8bcfd90874cbfd0d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:43.409096 712 leader_election.cc:343] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:43.412426 1157 ts_tablet_manager.cc:1918] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.412546 1157 log.cc:1198] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/75ec6b6b1ec1462d82008a78a885032d
I20250109 20:52:43.412947 1157 ts_tablet_manager.cc:1939] T 75ec6b6b1ec1462d82008a78a885032d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:43.414016 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 75ec6b6b1ec1462d82008a78a885032d (table default.Table_1d87425867fc462e86d1f51d3c7777dd [id=289e52afdcea437bb60cd657c8dc01d1]) successfully deleted
I20250109 20:52:43.417093 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 89fb6aeedc77452f99c46f03d6d8d089
I20250109 20:52:43.418087 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 with delete_type TABLET_DATA_DELETED (replaced by tablet 779153d4174b4887b5dc0bb355cb4a9d) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.418179 451 catalog_manager.cc:6596] ReplaceTablet: tablet 89fb6aeedc77452f99c46f03d6d8d089 deleted and replaced by tablet 779153d4174b4887b5dc0bb355cb4a9d
I20250109 20:52:43.418396 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 with delete_type TABLET_DATA_DELETED (replaced by tablet 779153d4174b4887b5dc0bb355cb4a9d) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:43.419049 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 89fb6aeedc77452f99c46f03d6d8d089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89fb6aeedc77452f99c46f03d6d8d089 already in progress: creating tablet
W20250109 20:52:43.421047 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 34cfa3005fdd40709a630d5bb80564db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34cfa3005fdd40709a630d5bb80564db already in progress: creating tablet
W20250109 20:52:43.421254 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ab00e387fbc843b2841bef2d3289df34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab00e387fbc843b2841bef2d3289df34 already in progress: creating tablet
W20250109 20:52:43.421447 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 175bf6c335a8405c97b296cf4718610b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 175bf6c335a8405c97b296cf4718610b already in progress: creating tablet
W20250109 20:52:43.421623 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 825efab5ff7b492ca3c7e3fe11ece098 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825efab5ff7b492ca3c7e3fe11ece098 already in progress: creating tablet
W20250109 20:52:43.421828 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 58817dddbe8749249a96c60dee41b520 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58817dddbe8749249a96c60dee41b520 already in progress: creating tablet
W20250109 20:52:43.422035 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 89fb6aeedc77452f99c46f03d6d8d089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89fb6aeedc77452f99c46f03d6d8d089 already in progress: creating tablet
I20250109 20:52:43.422490 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_60febb828b3c4979bf94fd0f01bdfe4c" } new_table_name: "default.Table_27fb907c2b28435db3522088b3d352fa" modify_external_catalogs: true
W20250109 20:52:43.424824 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 175bf6c335a8405c97b296cf4718610b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 175bf6c335a8405c97b296cf4718610b already in progress: creating tablet
W20250109 20:52:43.425218 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 58817dddbe8749249a96c60dee41b520 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58817dddbe8749249a96c60dee41b520 already in progress: creating tablet
I20250109 20:52:43.425407 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e03d934cb25e4e96b3c9670d7698253a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.426062 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e03d934cb25e4e96b3c9670d7698253a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.425818 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_31efa1aed4b840398b7d206e222ed31c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.426453 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31efa1aed4b840398b7d206e222ed31c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.425912 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 4f5a90e65e80494a8b451f2bc6baa2b1 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.427385 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c3dd2b1d3c0c4b20a2fe5f6533f750db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.427969 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3dd2b1d3c0c4b20a2fe5f6533f750db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.429239 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1bb496e68f2e45c39a6a4adcc8db8526" } new_table_name: "default.Table_e8ceeee244274a419a60eb82212c9d81" modify_external_catalogs: true
I20250109 20:52:43.432634 1187 raft_consensus.cc:491] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:43.432830 1187 raft_consensus.cc:513] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.433457 1187 leader_election.cc:290] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:43.433935 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38760b86fdfb4c8c8bcfd90874cbfd0d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:43.434183 1055 raft_consensus.cc:2463] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:43.435578 713 leader_election.cc:304] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:43.436017 1178 raft_consensus.cc:2798] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:43.436149 1178 raft_consensus.cc:491] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:43.436261 1178 raft_consensus.cc:3054] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:43.438802 1178 raft_consensus.cc:513] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.439806 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38760b86fdfb4c8c8bcfd90874cbfd0d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:43.440337 712 leader_election.cc:343] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:43.440682 1178 leader_election.cc:290] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:43.441687 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4f5a90e65e80494a8b451f2bc6baa2b1
I20250109 20:52:43.442154 453 catalog_manager.cc:6596] ReplaceTablet: tablet 4f5a90e65e80494a8b451f2bc6baa2b1 deleted and replaced by tablet b5ba73bbb08a4db0adfc9e72a84a22e8
I20250109 20:52:43.442368 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38760b86fdfb4c8c8bcfd90874cbfd0d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:43.442566 1055 raft_consensus.cc:3054] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:52:43.445022 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab593a602d61495eb7b323903ae610b6 consensus state has no leader
I20250109 20:52:43.445101 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5ba73bbb08a4db0adfc9e72a84a22e8) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.445399 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5ba73bbb08a4db0adfc9e72a84a22e8) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.425623 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_42c046d9a3f940728e68579acc2cd328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:43.446244 1055 raft_consensus.cc:2463] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
W20250109 20:52:43.445963 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f5a90e65e80494a8b451f2bc6baa2b1 already in progress: creating tablet
W20250109 20:52:43.446471 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42c046d9a3f940728e68579acc2cd328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:43.446656 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f5a90e65e80494a8b451f2bc6baa2b1 already in progress: creating tablet
I20250109 20:52:43.447124 713 leader_election.cc:304] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:43.447525 1178 raft_consensus.cc:2798] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:43.456952 1178 raft_consensus.cc:695] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
W20250109 20:52:43.457788 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68aa4a05c36e44b0aa6ee03780d2a815 consensus state has no leader
I20250109 20:52:43.457607 1178 consensus_queue.cc:237] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.457746 1231 raft_consensus.cc:491] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:43.458635 1231 raft_consensus.cc:513] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.461064 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e784a5378f7740baa39673071b56d8c5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:43.461316 1055 raft_consensus.cc:2463] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:43.461299 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c9d4e1c296d042c0be38c03d21165799" } new_table_name: "default.Table_418c798c8dbf4d3cad3c0e260ec69a65" modify_external_catalogs: true
I20250109 20:52:43.461493 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_354ff4ca71d54c2293728987a8eb1f88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.462016 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_354ff4ca71d54c2293728987a8eb1f88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.462903 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_47b199ac3a5945978c6af0a999fe64c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.463475 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47b199ac3a5945978c6af0a999fe64c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.463438 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5d6390d21a0241e1bd37f6158126ff0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.463979 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d6390d21a0241e1bd37f6158126ff0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.463977 713 leader_election.cc:304] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:43.465137 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_edf77c78245e46578d5b29f18ab58b40" } new_table_name: "default.Table_3565d86ee629490f99713f2b667ee3d2" modify_external_catalogs: true
I20250109 20:52:43.468128 1231 leader_election.cc:290] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:43.468736 1231 raft_consensus.cc:2798] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:43.469671 1176 tablet_bootstrap.cc:492] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.469942 1176 ts_tablet_manager.cc:1397] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.202s user 0.021s sys 0.016s
I20250109 20:52:43.471027 1176 raft_consensus.cc:357] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.471408 1176 raft_consensus.cc:383] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.471530 1176 raft_consensus.cc:738] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:52:43.471942 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32eb4dfcf96c4a05a55082e90ae04467 consensus state has no leader
I20250109 20:52:43.471858 1176 consensus_queue.cc:260] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.475033 1176 ts_tablet_manager.cc:1428] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:43.475687 1231 raft_consensus.cc:491] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:43.475894 1231 raft_consensus.cc:3054] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:43.476305 1176 tablet_bootstrap.cc:492] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.481341 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 with delete_type TABLET_DATA_DELETED (replaced by tablet 779153d4174b4887b5dc0bb355cb4a9d) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.483124 455 catalog_manager.cc:5526] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:43.483986 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]). Expected version 1 got 0
I20250109 20:52:43.485869 1231 raft_consensus.cc:513] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:43.486827 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c112bab5c5a42efab7582d83ca7eac1 has no consensus state
I20250109 20:52:43.486924 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e784a5378f7740baa39673071b56d8c5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:43.486333 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_63b84b4a77ba41198819de24f972ae17" } new_table_name: "default.Table_a4a01eeb3aa44fdc9926e35346c10db5" modify_external_catalogs: true
I20250109 20:52:43.487147 1055 raft_consensus.cc:3054] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:43.489861 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454b27918ab41ad9ab269185b2692fe with delete_type TABLET_DATA_DELETED (replaced by tablet 35d7cead4f4149ae8de48eee2cfd8679) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.490079 1166 tablet_replica.cc:331] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:43.490267 1166 raft_consensus.cc:2238] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.490430 1166 raft_consensus.cc:2267] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.490792 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 1044c212aada4df382d147c9a15ea4a7 (DEFAULT_TABLE table=default.Table_807cf97a78f84ba794b3d29b6e210081 [id=0eac4d9a91be4e02a4e766f780e4ca93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.491179 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1044c212aada4df382d147c9a15ea4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.491199 1055 raft_consensus.cc:2463] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:43.491504 713 leader_election.cc:304] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:43.491972 1176 tablet_bootstrap.cc:654] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.492995 1166 ts_tablet_manager.cc:1905] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.498608 1166 ts_tablet_manager.cc:1918] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.498762 1166 log.cc:1198] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/0454b27918ab41ad9ab269185b2692fe
I20250109 20:52:43.499118 1166 ts_tablet_manager.cc:1939] T 0454b27918ab41ad9ab269185b2692fe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:43.499444 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5ba73bbb08a4db0adfc9e72a84a22e8) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.500065 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 0454b27918ab41ad9ab269185b2692fe (table default.Table_bece8c2a3b4c4f42981b775b7b6ec654 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
W20250109 20:52:43.500401 964 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:41583 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:41583 (error 108)
W20250109 20:52:43.500564 989 connection.cc:537] client connection to 127.0.104.60:41583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:52:43.501056 1101 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:41583 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250109 20:52:43.501674 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1044c212aada4df382d147c9a15ea4a7 (DEFAULT_TABLE table=default.Table_807cf97a78f84ba794b3d29b6e210081 [id=0eac4d9a91be4e02a4e766f780e4ca93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.502012 1231 leader_election.cc:290] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:43.502059 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1044c212aada4df382d147c9a15ea4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.502197 1231 raft_consensus.cc:2798] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:43.502300 1231 raft_consensus.cc:695] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:43.502542 1231 consensus_queue.cc:237] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.505041 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e784a5378f7740baa39673071b56d8c5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:43.505285 919 raft_consensus.cc:2463] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:43.506228 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e784a5378f7740baa39673071b56d8c5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:43.506426 919 raft_consensus.cc:3054] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:43.348706 1165 consensus_queue.cc:260] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.508648 895 tablet_service.cc:1467] Processing CreateTablet for tablet 1044c212aada4df382d147c9a15ea4a7 (DEFAULT_TABLE table=default.Table_807cf97a78f84ba794b3d29b6e210081 [id=0eac4d9a91be4e02a4e766f780e4ca93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.509037 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1044c212aada4df382d147c9a15ea4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.509269 919 raft_consensus.cc:2463] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
W20250109 20:52:43.510219 823 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:41583 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:41583 (error 108)
I20250109 20:52:43.510432 1176 tablet_bootstrap.cc:492] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.510587 1176 ts_tablet_manager.cc:1397] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
W20250109 20:52:43.515332 1207 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:52:43.500958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:41583 (local address 127.0.104.2:41221)
0109 20:52:43.501192 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0109 20:52:43.501210 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0109 20:52:43.501313 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0109 20:52:43.514564 (+ 13251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133}
W20250109 20:52:43.515532 1237 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:52:43.501805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:41583 (local address 127.0.104.3:41753)
0109 20:52:43.514502 (+ 12697us) negotiation.cc:107] Waiting for socket to connect
0109 20:52:43.514688 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":12552,"thread_start_us":12778,"threads_started":1}
I20250109 20:52:43.517544 1165 ts_tablet_manager.cc:1428] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.170s user 0.003s sys 0.000s
I20250109 20:52:43.518719 1033 tablet_service.cc:1467] Processing CreateTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e (DEFAULT_TABLE table=default.Table_82b8503333e64f94829491f22a40b92b [id=87109c5773c14f159f596b9d30844948]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.519130 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f865acd2c9aa47ec8ebc3bf90d81c21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.518966 1176 raft_consensus.cc:357] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.519265 1176 raft_consensus.cc:383] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.519371 1176 raft_consensus.cc:738] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.519687 1176 consensus_queue.cc:260] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.521703 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38760b86fdfb4c8c8bcfd90874cbfd0d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:43.522660 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd067c335354357b44dac433f8c1523" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:43.522890 778 raft_consensus.cc:2463] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:43.523939 1176 ts_tablet_manager.cc:1428] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250109 20:52:43.524019 990 leader_election.cc:304] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:43.524734 1187 raft_consensus.cc:2798] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:43.524842 1187 raft_consensus.cc:491] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:43.524956 1187 raft_consensus.cc:3054] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:43.525501 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 58817dddbe8749249a96c60dee41b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.526381 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5ab3d2823bee4f739f414db53a1a6cd9" } modify_external_catalogs: true
I20250109 20:52:43.526556 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5ab3d2823bee4f739f414db53a1a6cd9" } modify_external_catalogs: true
I20250109 20:52:43.528118 1187 raft_consensus.cc:513] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.528970 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_42c046d9a3f940728e68579acc2cd328" } modify_external_catalogs: true
I20250109 20:52:43.529140 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_42c046d9a3f940728e68579acc2cd328" } modify_external_catalogs: true
I20250109 20:52:43.529294 1187 leader_election.cc:290] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:43.530016 1166 tablet_replica.cc:331] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:43.529944 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd067c335354357b44dac433f8c1523" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:43.530161 778 raft_consensus.cc:3054] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:43.530190 1166 raft_consensus.cc:2238] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.530334 1166 raft_consensus.cc:2267] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:43.530642 1236 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:52:43.510857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:41583 (local address 127.0.104.1:59529)
0109 20:52:43.525125 (+ 14268us) negotiation.cc:107] Waiting for socket to connect
0109 20:52:43.525286 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":14167,"threads_started":1}
I20250109 20:52:43.530622 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_82b8503333e64f94829491f22a40b92b" } modify_external_catalogs: true
I20250109 20:52:43.530769 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_82b8503333e64f94829491f22a40b92b" } modify_external_catalogs: true
I20250109 20:52:43.530923 1166 ts_tablet_manager.cc:1905] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.531682 1191 raft_consensus.cc:491] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:43.531832 1191 raft_consensus.cc:513] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.532392 1191 leader_election.cc:290] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:43.533166 778 raft_consensus.cc:2463] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:43.533627 990 leader_election.cc:304] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:43.533427 448 catalog_manager.cc:5526] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:43.533921 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_82b978780ede49bfaa599c5af6e7d57d [id=e4a51641b240493d9f69b9031ef5d4b1]). Expected version 2 got 0
I20250109 20:52:43.533995 1187 raft_consensus.cc:2798] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:43.536582 1166 ts_tablet_manager.cc:1918] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:43.536918 854 leader_election.cc:343] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 103722f5a2314bbeb9b141bb0a460ece
I20250109 20:52:43.538372 1166 log.cc:1198] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/58817dddbe8749249a96c60dee41b520
I20250109 20:52:43.538771 1166 ts_tablet_manager.cc:1939] T 58817dddbe8749249a96c60dee41b520 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:43.539017 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9214f432e5c04dca831c8a093b699430" } modify_external_catalogs: true
I20250109 20:52:43.539152 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9214f432e5c04dca831c8a093b699430" } modify_external_catalogs: true
I20250109 20:52:43.539289 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 58817dddbe8749249a96c60dee41b520 (table default.Table_d0a7a37106ea46e683ead0b81aa0585e [id=fc46d1a61d7d4769bfcf249712d2e2b9]) successfully deleted
I20250109 20:52:43.541802 1176 tablet_bootstrap.cc:492] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.543648 1176 tablet_bootstrap.cc:654] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.538390 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103722f5a2314bbeb9b141bb0a460ece" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:43.546221 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 01e59a5830964cc2901b0b8956d0be21
I20250109 20:52:43.546190 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38760b86fdfb4c8c8bcfd90874cbfd0d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:43.546414 854 leader_election.cc:343] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:43.546557 854 leader_election.cc:304] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:43.546728 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd067c335354357b44dac433f8c1523" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:43.546941 1191 raft_consensus.cc:2743] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:43.546916 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_27fb907c2b28435db3522088b3d352fa" } modify_external_catalogs: true
I20250109 20:52:43.547075 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_27fb907c2b28435db3522088b3d352fa" } modify_external_catalogs: true
I20250109 20:52:43.547154 919 raft_consensus.cc:2463] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:43.547253 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd067c335354357b44dac433f8c1523" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:43.547497 918 raft_consensus.cc:3054] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:43.548200 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 58817dddbe8749249a96c60dee41b520 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.550976 918 raft_consensus.cc:2463] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
W20250109 20:52:43.552273 989 leader_election.cc:343] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:43.553438 989 leader_election.cc:304] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:43.554081 1187 raft_consensus.cc:695] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:43.554991 1187 consensus_queue.cc:237] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.558310 1240 raft_consensus.cc:2743] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:43.559288 1158 tablet_bootstrap.cc:492] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.561345 1158 ts_tablet_manager.cc:1397] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.268s user 0.084s sys 0.136s
I20250109 20:52:43.562261 1158 raft_consensus.cc:357] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.562520 1158 raft_consensus.cc:383] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.562598 1158 raft_consensus.cc:738] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.562865 1158 consensus_queue.cc:260] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.566043 1176 tablet_bootstrap.cc:492] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.566179 1176 ts_tablet_manager.cc:1397] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250109 20:52:43.566303 1158 ts_tablet_manager.cc:1428] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:43.566610 1158 tablet_bootstrap.cc:492] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.566919 1176 raft_consensus.cc:357] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.567131 1176 raft_consensus.cc:383] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.567212 1176 raft_consensus.cc:738] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.567412 1176 consensus_queue.cc:260] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.567862 1176 ts_tablet_manager.cc:1428] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:43.568109 1176 tablet_bootstrap.cc:492] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.568770 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 68aa4a05c36e44b0aa6ee03780d2a815
I20250109 20:52:43.569800 1176 tablet_bootstrap.cc:654] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.569793 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6bea28fb446e49669133e21b02c416ad
I20250109 20:52:43.569868 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.570394 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bea28fb446e49669133e21b02c416ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:43.570422 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 68aa4a05c36e44b0aa6ee03780d2a815 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68aa4a05c36e44b0aa6ee03780d2a815 already in progress: creating tablet
I20250109 20:52:43.570575 1157 tablet_replica.cc:331] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:43.570739 1157 raft_consensus.cc:2238] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.570873 1157 raft_consensus.cc:2267] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.571420 1157 ts_tablet_manager.cc:1905] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.571712 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bea28fb446e49669133e21b02c416ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:43.572280 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6bea28fb446e49669133e21b02c416ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bea28fb446e49669133e21b02c416ad already in progress: creating tablet
I20250109 20:52:43.572804 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet c86d45bcdd084f798a437e4587761809
I20250109 20:52:43.574802 1158 tablet_bootstrap.cc:654] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.577661 1157 ts_tablet_manager.cc:1918] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.577847 1157 log.cc:1198] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6bea28fb446e49669133e21b02c416ad
I20250109 20:52:43.578238 1157 ts_tablet_manager.cc:1939] T 6bea28fb446e49669133e21b02c416ad P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:43.579299 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6bea28fb446e49669133e21b02c416ad (table default.Table_5ab3d2823bee4f739f414db53a1a6cd9 [id=06cecc207bb746dd9b0cb0f6eac4017e]) successfully deleted
W20250109 20:52:43.579723 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed4866f6007d411da5cf4832fa92ece9 consensus state has no leader
I20250109 20:52:43.570865 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.582914 1165 tablet_bootstrap.cc:492] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:43.584676 1165 tablet_bootstrap.cc:654] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:43.581722 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 68aa4a05c36e44b0aa6ee03780d2a815 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68aa4a05c36e44b0aa6ee03780d2a815 already in progress: creating tablet
W20250109 20:52:43.565802 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 89fb6aeedc77452f99c46f03d6d8d089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89fb6aeedc77452f99c46f03d6d8d089 already in progress: creating tablet
W20250109 20:52:43.585924 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f5a90e65e80494a8b451f2bc6baa2b1 already in progress: creating tablet
I20250109 20:52:43.586624 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 103722f5a2314bbeb9b141bb0a460ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.586730 1176 tablet_bootstrap.cc:492] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.586844 1176 ts_tablet_manager.cc:1397] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250109 20:52:43.587038 1160 tablet_replica.cc:331] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:43.587230 1160 raft_consensus.cc:2238] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.587538 1160 raft_consensus.cc:2267] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.587527 1176 raft_consensus.cc:357] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.587713 1176 raft_consensus.cc:383] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.587796 1176 raft_consensus.cc:738] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.588383 888 tablet_service.cc:1467] Processing CreateTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e (DEFAULT_TABLE table=default.Table_82b8503333e64f94829491f22a40b92b [id=87109c5773c14f159f596b9d30844948]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.588536 1160 ts_tablet_manager.cc:1905] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.588815 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f865acd2c9aa47ec8ebc3bf90d81c21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.588837 1176 consensus_queue.cc:260] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.589596 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 6acadf60d566407aabc1a2395d1d235f (DEFAULT_TABLE table=default.Table_08e4b27362104ad4af8816b4facabed7 [id=66c37d69f6bb4fe382c332f5c04e7a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.590023 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acadf60d566407aabc1a2395d1d235f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.592551 751 tablet_service.cc:1467] Processing CreateTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e (DEFAULT_TABLE table=default.Table_82b8503333e64f94829491f22a40b92b [id=87109c5773c14f159f596b9d30844948]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.593408 1176 ts_tablet_manager.cc:1428] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:52:43.593703 1176 tablet_bootstrap.cc:492] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.593817 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f865acd2c9aa47ec8ebc3bf90d81c21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.595340 1176 tablet_bootstrap.cc:654] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.598081 1160 ts_tablet_manager.cc:1918] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.598242 1160 log.cc:1198] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/103722f5a2314bbeb9b141bb0a460ece
I20250109 20:52:43.598699 1160 ts_tablet_manager.cc:1939] T 103722f5a2314bbeb9b141bb0a460ece P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:43.599155 754 tablet_service.cc:1467] Processing CreateTablet for tablet 6acadf60d566407aabc1a2395d1d235f (DEFAULT_TABLE table=default.Table_08e4b27362104ad4af8816b4facabed7 [id=66c37d69f6bb4fe382c332f5c04e7a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.599222 453 master_service.cc:946] ReplaceTablet: received request to replace tablet ded8e3a447c84b9b8ec4ccd70199ccd5 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.599498 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acadf60d566407aabc1a2395d1d235f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.601760 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 (DEFAULT_TABLE table=default.Table_0504385486ba4dd18a9a56a3f206aa9a [id=a79a90b8e83e4b4a80cb0e2a8693d7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.602067 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a54393ff3264c52bfdeee2cb9046828. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:43.608733 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 -> Peer 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583): Couldn't send request to peer 36a29ab2002d4eb389cd11778988e1e9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:43.609611 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 103722f5a2314bbeb9b141bb0a460ece (table default.Table_0710fdbfdd7a4e7b892b501e77fc2e72 [id=606e0f04684243e8a30813edb6e96766]) successfully deleted
I20250109 20:52:43.610268 899 tablet_service.cc:1467] Processing CreateTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 (DEFAULT_TABLE table=default.Table_0504385486ba4dd18a9a56a3f206aa9a [id=a79a90b8e83e4b4a80cb0e2a8693d7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.610591 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a54393ff3264c52bfdeee2cb9046828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.610720 754 tablet_service.cc:1467] Processing CreateTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 (DEFAULT_TABLE table=default.Table_0504385486ba4dd18a9a56a3f206aa9a [id=a79a90b8e83e4b4a80cb0e2a8693d7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.611050 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a54393ff3264c52bfdeee2cb9046828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.616010 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6acadf60d566407aabc1a2395d1d235f (DEFAULT_TABLE table=default.Table_08e4b27362104ad4af8816b4facabed7 [id=66c37d69f6bb4fe382c332f5c04e7a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.616325 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acadf60d566407aabc1a2395d1d235f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.620903 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bea28fb446e49669133e21b02c416ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.621438 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:43.621444 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6bea28fb446e49669133e21b02c416ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bea28fb446e49669133e21b02c416ad already in progress: creating tablet
W20250109 20:52:43.622028 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 68aa4a05c36e44b0aa6ee03780d2a815 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68aa4a05c36e44b0aa6ee03780d2a815 already in progress: creating tablet
I20250109 20:52:43.622881 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cabbd075716c4d18aba39e549c7f0043"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.623450 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cabbd075716c4d18aba39e549c7f0043 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.624835 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.625423 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.630098 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 68aa4a05c36e44b0aa6ee03780d2a815 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.630873 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6c5282bf89ae4261a99e8c8b6d3b0e95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.631359 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c5282bf89ae4261a99e8c8b6d3b0e95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.631395 1158 tablet_bootstrap.cc:492] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.631583 1158 ts_tablet_manager.cc:1397] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.065s user 0.017s sys 0.009s
I20250109 20:52:43.632457 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ded8e3a447c84b9b8ec4ccd70199ccd5
I20250109 20:52:43.632557 1158 raft_consensus.cc:357] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.632962 1158 raft_consensus.cc:383] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.633069 1158 raft_consensus.cc:738] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.633214 896 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1925a82eb711409ab2c5429c61edf9d8) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.633399 1158 consensus_queue.cc:260] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:43.633934 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded8e3a447c84b9b8ec4ccd70199ccd5 already in progress: creating tablet
I20250109 20:52:43.635049 453 catalog_manager.cc:6596] ReplaceTablet: tablet ded8e3a447c84b9b8ec4ccd70199ccd5 deleted and replaced by tablet 1925a82eb711409ab2c5429c61edf9d8
I20250109 20:52:43.636504 754 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1925a82eb711409ab2c5429c61edf9d8) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.636598 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1925a82eb711409ab2c5429c61edf9d8) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.636978 1158 ts_tablet_manager.cc:1428] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250109 20:52:43.637243 1158 tablet_bootstrap.cc:492] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:43.637223 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded8e3a447c84b9b8ec4ccd70199ccd5 already in progress: creating tablet
W20250109 20:52:43.637413 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded8e3a447c84b9b8ec4ccd70199ccd5 already in progress: creating tablet
I20250109 20:52:43.637430 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f865acd2c9aa47ec8ebc3bf90d81c21e
I20250109 20:52:43.638782 1158 tablet_bootstrap.cc:654] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.639029 1240 raft_consensus.cc:491] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:43.639174 1240 raft_consensus.cc:513] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.640130 1240 leader_election.cc:290] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:43.640671 1176 tablet_bootstrap.cc:492] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.640828 1176 ts_tablet_manager.cc:1397] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.012s
I20250109 20:52:43.640862 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a1a6a0a1f4411aa0b57481b6291808" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:43.641531 1176 raft_consensus.cc:357] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.641739 1176 raft_consensus.cc:383] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.641810 1176 raft_consensus.cc:738] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.641820 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a1a6a0a1f4411aa0b57481b6291808" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:43.643635 754 tablet_service.cc:1467] Processing CreateTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b (DEFAULT_TABLE table=default.Table_51da7d20722144bf824bdc0388d68808 [id=92af12dca25846f694464bd77a74c1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.644050 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b47ee787ca4dc5a4a56ae582d15b8b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:43.645135 990 leader_election.cc:343] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:43.646270 888 tablet_service.cc:1467] Processing CreateTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b (DEFAULT_TABLE table=default.Table_51da7d20722144bf824bdc0388d68808 [id=92af12dca25846f694464bd77a74c1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.646709 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b47ee787ca4dc5a4a56ae582d15b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.649075 895 tablet_service.cc:1467] Processing CreateTablet for tablet ed4866f6007d411da5cf4832fa92ece9 (DEFAULT_TABLE table=default.Table_a4a01eeb3aa44fdc9926e35346c10db5 [id=b494cc0e89104a849325f838046bc82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.649500 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4866f6007d411da5cf4832fa92ece9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.651288 1165 tablet_bootstrap.cc:492] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:43.651455 1165 ts_tablet_manager.cc:1397] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.003s
I20250109 20:52:43.652180 1165 raft_consensus.cc:357] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.652427 1165 raft_consensus.cc:383] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.652511 1165 raft_consensus.cc:738] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.653151 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ed4866f6007d411da5cf4832fa92ece9 (DEFAULT_TABLE table=default.Table_a4a01eeb3aa44fdc9926e35346c10db5 [id=b494cc0e89104a849325f838046bc82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.653723 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 03e06e64448a4fb2bf41fc6e3d0361a6 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.654769 751 tablet_service.cc:1467] Processing CreateTablet for tablet ed4866f6007d411da5cf4832fa92ece9 (DEFAULT_TABLE table=default.Table_a4a01eeb3aa44fdc9926e35346c10db5 [id=b494cc0e89104a849325f838046bc82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.655210 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4866f6007d411da5cf4832fa92ece9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:43.657079 989 leader_election.cc:343] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:43.657250 989 leader_election.cc:304] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:43.658712 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4866f6007d411da5cf4832fa92ece9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.658977 1176 consensus_queue.cc:260] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.659487 1187 raft_consensus.cc:2743] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:43.660025 1176 ts_tablet_manager.cc:1428] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20250109 20:52:43.660048 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.660306 1176 tablet_bootstrap.cc:492] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.660692 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2cb493640d614e088e638e580754f9d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.661206 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cb493640d614e088e638e580754f9d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.662102 1176 tablet_bootstrap.cc:654] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.664242 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.664796 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_71377d92f20d4047a2af17e649b9ac62" } new_table_name: "default.Table_9016eedbbe184e968d3a7d60d2286789" modify_external_catalogs: true
W20250109 20:52:43.665907 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f865acd2c9aa47ec8ebc3bf90d81c21e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f865acd2c9aa47ec8ebc3bf90d81c21e already in progress: creating tablet
I20250109 20:52:43.668100 1198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:43.668306 1198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: LEADER
I20250109 20:52:43.673725 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_055a7c502de346ff92416671164dc684"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.674326 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_055a7c502de346ff92416671164dc684 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.675974 455 master_service.cc:946] ReplaceTablet: received request to replace tablet ded8e3a447c84b9b8ec4ccd70199ccd5 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.672868 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1e48be0e18b145708a9b6fa28ae21112"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.677706 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e48be0e18b145708a9b6fa28ae21112 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.678648 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c3c448aaf88a40de8c2d09f88e78442f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.681233 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3c448aaf88a40de8c2d09f88e78442f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.681100 1165 consensus_queue.cc:260] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.681774 1165 ts_tablet_manager.cc:1428] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20250109 20:52:43.679385 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fd20b680db98408db7e7f0278832db97" } new_table_name: "default.Table_922405f1ef264d2a9457f48176a918a6" modify_external_catalogs: true
I20250109 20:52:43.684990 1165 tablet_bootstrap.cc:492] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:43.686959 1165 tablet_bootstrap.cc:654] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.688835 896 tablet_service.cc:1514] Processing DeleteTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:43.689742 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f865acd2c9aa47ec8ebc3bf90d81c21e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f865acd2c9aa47ec8ebc3bf90d81c21e already in progress: creating tablet
W20250109 20:52:43.679484 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f865acd2c9aa47ec8ebc3bf90d81c21e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f865acd2c9aa47ec8ebc3bf90d81c21e already in progress: creating tablet
I20250109 20:52:43.679711 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_41c33f8ca7514962afbb30abb1e704da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.690440 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41c33f8ca7514962afbb30abb1e704da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.679831 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_418c798c8dbf4d3cad3c0e260ec69a65" } new_table_name: "default.Table_384c816ede6b4c558524a02495bb3dc0" modify_external_catalogs: true
I20250109 20:52:43.700901 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b (DEFAULT_TABLE table=default.Table_51da7d20722144bf824bdc0388d68808 [id=92af12dca25846f694464bd77a74c1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.701534 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b47ee787ca4dc5a4a56ae582d15b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.714159 446 catalog_manager.cc:5526] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
I20250109 20:52:43.714743 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet cdd067c335354357b44dac433f8c1523 (table default.Table_fd20b680db98408db7e7f0278832db97 [id=8c860f8c96254692906f892a25530962]). Expected version 3 got 0
I20250109 20:52:43.717288 1165 tablet_bootstrap.cc:492] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:43.717417 1165 ts_tablet_manager.cc:1397] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.002s
I20250109 20:52:43.717361 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7541b9bb11c44022ae972d09079fd75d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:43.717767 1176 tablet_bootstrap.cc:492] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
W20250109 20:52:43.717777 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7541b9bb11c44022ae972d09079fd75d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.717888 1176 ts_tablet_manager.cc:1397] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.013s
I20250109 20:52:43.718102 1165 raft_consensus.cc:357] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.718286 1165 raft_consensus.cc:383] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.718360 1165 raft_consensus.cc:738] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.718550 1165 consensus_queue.cc:260] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.718569 1176 raft_consensus.cc:357] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.718739 1176 raft_consensus.cc:383] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.718824 1176 raft_consensus.cc:738] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.719007 1165 ts_tablet_manager.cc:1428] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:52:43.719034 1176 consensus_queue.cc:260] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.719231 1165 tablet_bootstrap.cc:492] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:43.719481 1176 ts_tablet_manager.cc:1428] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:43.719715 1176 tablet_bootstrap.cc:492] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.720769 1165 tablet_bootstrap.cc:654] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.721109 1176 tablet_bootstrap.cc:654] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.726625 888 tablet_service.cc:1514] Processing DeleteTablet for tablet be88a4cf79c14c0fb3ab1110a3fa1714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.726953 1160 tablet_replica.cc:331] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:43.727164 1160 raft_consensus.cc:2238] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.727344 1160 raft_consensus.cc:2267] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.727747 1191 raft_consensus.cc:491] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:43.727916 1191 raft_consensus.cc:513] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.728121 1160 ts_tablet_manager.cc:1905] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.728241 1158 tablet_bootstrap.cc:492] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.728381 1158 ts_tablet_manager.cc:1397] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.091s user 0.034s sys 0.038s
I20250109 20:52:43.728734 1191 leader_election.cc:290] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:43.729154 1158 raft_consensus.cc:357] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.729378 1158 raft_consensus.cc:383] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.729454 1158 raft_consensus.cc:738] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.729673 1158 consensus_queue.cc:260] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.730154 1158 ts_tablet_manager.cc:1428] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:43.730405 1158 tablet_bootstrap.cc:492] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:43.730360 854 leader_election.cc:343] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 75ec6b6b1ec1462d82008a78a885032d
W20250109 20:52:43.730787 854 leader_election.cc:343] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 75ec6b6b1ec1462d82008a78a885032d
I20250109 20:52:43.730906 854 leader_election.cc:304] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:43.731794 1191 raft_consensus.cc:2743] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:43.731913 1158 tablet_bootstrap.cc:654] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.716621 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1838fc762deb4f71b0fd91a2eaa397ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.734784 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1838fc762deb4f71b0fd91a2eaa397ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.736610 1160 ts_tablet_manager.cc:1918] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.736804 1160 log.cc:1198] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/be88a4cf79c14c0fb3ab1110a3fa1714
I20250109 20:52:43.737236 1160 ts_tablet_manager.cc:1939] T be88a4cf79c14c0fb3ab1110a3fa1714 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:43.738215 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet be88a4cf79c14c0fb3ab1110a3fa1714 (table default.Table_1b67d2a3c0b3403b8041cfb32c37b227 [id=e68ca2b1501a493a97b5ae61bdd9fe60]) successfully deleted
W20250109 20:52:43.740922 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdd067c335354357b44dac433f8c1523 consensus state has no leader
I20250109 20:52:43.743139 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49d1907147ef4990876347467005e31e" } new_table_name: "default.Table_609a0ae37cfd4b6d8939b4829beb84d6" modify_external_catalogs: true
I20250109 20:52:43.745363 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ec6b6b1ec1462d82008a78a885032d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.748903 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 58817dddbe8749249a96c60dee41b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.749228 1157 tablet_replica.cc:331] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:43.749450 1157 raft_consensus.cc:2238] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.749639 1157 raft_consensus.cc:2267] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.750121 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_055a7c502de346ff92416671164dc684" } modify_external_catalogs: true
I20250109 20:52:43.750309 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_055a7c502de346ff92416671164dc684" } modify_external_catalogs: true
I20250109 20:52:43.750497 1157 ts_tablet_manager.cc:1905] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.754855 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 03e06e64448a4fb2bf41fc6e3d0361a6
W20250109 20:52:43.755571 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32eb4dfcf96c4a05a55082e90ae04467 consensus state has no leader
I20250109 20:52:43.755744 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 with delete_type TABLET_DATA_DELETED (replaced by tablet cba0ffb1b394431ba30918911cdee7b7) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.756491 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 with delete_type TABLET_DATA_DELETED (replaced by tablet cba0ffb1b394431ba30918911cdee7b7) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.757272 453 catalog_manager.cc:6596] ReplaceTablet: tablet 03e06e64448a4fb2bf41fc6e3d0361a6 deleted and replaced by tablet cba0ffb1b394431ba30918911cdee7b7
I20250109 20:52:43.757335 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 with delete_type TABLET_DATA_DELETED (replaced by tablet cba0ffb1b394431ba30918911cdee7b7) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.757562 1160 tablet_replica.cc:331] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:43.757755 1160 raft_consensus.cc:2238] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.757927 1160 raft_consensus.cc:2267] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:43.758003 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03e06e64448a4fb2bf41fc6e3d0361a6 already in progress: creating tablet
W20250109 20:52:43.758296 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1925a82eb711409ab2c5429c61edf9d8 has no consensus state
I20250109 20:52:43.758565 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d with delete_type TABLET_DATA_DELETED (replaced by tablet 66c6016d88b54eb390f880945ddb1ea6) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.758850 1160 ts_tablet_manager.cc:1905] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.759661 1157 ts_tablet_manager.cc:1918] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.759769 1157 log.cc:1198] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/58817dddbe8749249a96c60dee41b520
I20250109 20:52:43.760090 1157 ts_tablet_manager.cc:1939] T 58817dddbe8749249a96c60dee41b520 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:43.760818 1176 tablet_bootstrap.cc:492] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.760970 1176 ts_tablet_manager.cc:1397] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.004s
I20250109 20:52:43.761718 1176 raft_consensus.cc:357] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.761960 1176 raft_consensus.cc:383] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.762048 1176 raft_consensus.cc:738] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.762300 1176 consensus_queue.cc:260] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.764142 1176 ts_tablet_manager.cc:1428] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:43.764463 1176 tablet_bootstrap.cc:492] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.766683 1176 tablet_bootstrap.cc:654] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.767446 1165 tablet_bootstrap.cc:492] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:43.767583 1165 ts_tablet_manager.cc:1397] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.048s user 0.026s sys 0.011s
I20250109 20:52:43.767705 1157 tablet_replica.cc:331] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:43.767910 1157 raft_consensus.cc:2238] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.768046 1157 raft_consensus.cc:2267] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.768242 1165 raft_consensus.cc:357] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.768425 1165 raft_consensus.cc:383] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.768462 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 75ec6b6b1ec1462d82008a78a885032d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.768507 1165 raft_consensus.cc:738] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.768831 1165 consensus_queue.cc:260] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.769116 1158 tablet_bootstrap.cc:492] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.769243 1158 ts_tablet_manager.cc:1397] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.002s
I20250109 20:52:43.769937 1158 raft_consensus.cc:357] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.770155 1158 raft_consensus.cc:383] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.770248 1158 raft_consensus.cc:738] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.770488 1158 consensus_queue.cc:260] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.770982 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 58817dddbe8749249a96c60dee41b520 (table default.Table_d0a7a37106ea46e683ead0b81aa0585e [id=fc46d1a61d7d4769bfcf249712d2e2b9]) successfully deleted
W20250109 20:52:43.771239 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03e06e64448a4fb2bf41fc6e3d0361a6 already in progress: creating tablet
I20250109 20:52:43.771754 1165 ts_tablet_manager.cc:1428] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250109 20:52:43.774024 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cba0ffb1b394431ba30918911cdee7b7 has no consensus state
I20250109 20:52:43.774582 1165 tablet_bootstrap.cc:492] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:43.775646 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.775700 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.775902 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.776100 1166 tablet_replica.cc:331] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:43.776290 1166 raft_consensus.cc:2238] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.776449 1166 raft_consensus.cc:2267] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.777678 1166 ts_tablet_manager.cc:1905] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.778299 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0504385486ba4dd18a9a56a3f206aa9a" } modify_external_catalogs: true
I20250109 20:52:43.778499 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0504385486ba4dd18a9a56a3f206aa9a" } modify_external_catalogs: true
I20250109 20:52:43.778784 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]). Expected version 1 got 0
I20250109 20:52:43.781991 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 103722f5a2314bbeb9b141bb0a460ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.781996 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_807cf97a78f84ba794b3d29b6e210081" } modify_external_catalogs: true
I20250109 20:52:43.782138 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_807cf97a78f84ba794b3d29b6e210081" } modify_external_catalogs: true
I20250109 20:52:43.782966 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]). Expected version 1 got 0
I20250109 20:52:43.784036 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d with delete_type TABLET_DATA_DELETED (replaced by tablet 66c6016d88b54eb390f880945ddb1ea6) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.785216 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.785527 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 58817dddbe8749249a96c60dee41b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.785840 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c5282bf89ae4261a99e8c8b6d3b0e95" } modify_external_catalogs: true
I20250109 20:52:43.786001 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c5282bf89ae4261a99e8c8b6d3b0e95" } modify_external_catalogs: true
I20250109 20:52:43.789011 1157 ts_tablet_manager.cc:1905] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.789669 1166 ts_tablet_manager.cc:1918] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.790050 1166 log.cc:1198] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f6a1a6a0a1f4411aa0b57481b6291808
I20250109 20:52:43.790573 1166 ts_tablet_manager.cc:1939] T f6a1a6a0a1f4411aa0b57481b6291808 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:43.791174 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d336fdcf1ddc45638046b11a15ac7f59
I20250109 20:52:43.792305 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f6a1a6a0a1f4411aa0b57481b6291808 (table default.Table_dab2393d93b14ec79d96807a37360e33 [id=fcec4978e3584a9cabd5641b5ffb7340]) successfully deleted
I20250109 20:52:43.792526 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_82b978780ede49bfaa599c5af6e7d57d" } new_table_name: "default.Table_2acf186aebe7415881f4304654dc6a83" modify_external_catalogs: true
I20250109 20:52:43.793692 1166 tablet_replica.cc:331] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:43.794235 1166 raft_consensus.cc:2238] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.794653 1166 raft_consensus.cc:2267] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.795523 1166 ts_tablet_manager.cc:1905] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.795621 1157 ts_tablet_manager.cc:1918] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.795766 1157 log.cc:1198] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/e5d6dc79bf1e48ee9cf2bbb70463741d
I20250109 20:52:43.796109 1157 ts_tablet_manager.cc:1939] T e5d6dc79bf1e48ee9cf2bbb70463741d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:43.796757 1165 tablet_bootstrap.cc:654] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.797094 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet e5d6dc79bf1e48ee9cf2bbb70463741d (table default.Table_f9f28decdfb64455809f7b283552a4a5 [id=ae7fb9326c2c49a593d1111d45ea424f]) successfully deleted
W20250109 20:52:43.797485 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 26f5b4af510942c8b50838df3bc8d8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26f5b4af510942c8b50838df3bc8d8fc already in progress: creating tablet
I20250109 20:52:43.798106 1160 ts_tablet_manager.cc:1918] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:43.798398 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 58817dddbe8749249a96c60dee41b520 because the tablet was not found. No further retry: Not found: Tablet not found: 58817dddbe8749249a96c60dee41b520
I20250109 20:52:43.798758 1160 log.cc:1198] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/75ec6b6b1ec1462d82008a78a885032d
I20250109 20:52:43.799055 1158 ts_tablet_manager.cc:1428] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250109 20:52:43.799449 1160 ts_tablet_manager.cc:1939] T 75ec6b6b1ec1462d82008a78a885032d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:43.801103 1158 tablet_bootstrap.cc:492] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.802512 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet a70647c6debd447e86114a37fd203f65
I20250109 20:52:43.802845 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 75ec6b6b1ec1462d82008a78a885032d (table default.Table_1d87425867fc462e86d1f51d3c7777dd [id=289e52afdcea437bb60cd657c8dc01d1]) successfully deleted
W20250109 20:52:43.803053 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03e06e64448a4fb2bf41fc6e3d0361a6 already in progress: creating tablet
I20250109 20:52:43.803498 888 tablet_service.cc:1467] Processing CreateTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 (DEFAULT_TABLE table=default.Table_5d6390d21a0241e1bd37f6158126ff0e [id=7bd35efd3fad42dab4f6aa66d998c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.803953 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4f020c5ba643aeb11f8e08d748be46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.804658 1158 tablet_bootstrap.cc:654] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.806398 1165 tablet_bootstrap.cc:492] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:43.806524 1165 ts_tablet_manager.cc:1397] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20250109 20:52:43.807235 1165 raft_consensus.cc:357] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.807467 1165 raft_consensus.cc:383] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.807554 1165 raft_consensus.cc:738] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.807680 895 tablet_service.cc:1467] Processing CreateTablet for tablet c528a1a9ae0541ba9d4a80e38e743969 (DEFAULT_TABLE table=default.Table_31efa1aed4b840398b7d206e222ed31c [id=c96aada642b84c20894926b34fbea386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.808058 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c528a1a9ae0541ba9d4a80e38e743969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.809473 1165 consensus_queue.cc:260] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.810745 750 tablet_service.cc:1467] Processing CreateTablet for tablet 4c112bab5c5a42efab7582d83ca7eac1 (DEFAULT_TABLE table=default.Table_3565d86ee629490f99713f2b667ee3d2 [id=b7903ec3ae3b4d0f8ef509c4c05f11f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.810951 758 tablet_service.cc:1467] Processing CreateTablet for tablet da2d6dda69da4a59b055106c1041ec2b (DEFAULT_TABLE table=default.Table_c3dd2b1d3c0c4b20a2fe5f6533f750db [id=ce415eef7536445b99cab5e0769d359b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.811091 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c112bab5c5a42efab7582d83ca7eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.811326 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2d6dda69da4a59b055106c1041ec2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.813163 747 tablet_service.cc:1467] Processing CreateTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 (DEFAULT_TABLE table=default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.813745 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1925a82eb711409ab2c5429c61edf9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.815161 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 (DEFAULT_TABLE table=default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee [id=00e9e64762824d89b0a757e997078b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.815312 1165 ts_tablet_manager.cc:1428] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250109 20:52:43.815654 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa23057a60a4a3eb7755e799bc3b911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.817232 751 tablet_service.cc:1467] Processing CreateTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 (DEFAULT_TABLE table=default.Table_5d6390d21a0241e1bd37f6158126ff0e [id=7bd35efd3fad42dab4f6aa66d998c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.817623 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4f020c5ba643aeb11f8e08d748be46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.819790 1019 tablet_service.cc:1467] Processing CreateTablet for tablet f1589b79a56148b6ab2946943de39fb2 (DEFAULT_TABLE table=default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.820327 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 (DEFAULT_TABLE table=default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.820542 754 tablet_service.cc:1467] Processing CreateTablet for tablet c528a1a9ae0541ba9d4a80e38e743969 (DEFAULT_TABLE table=default.Table_31efa1aed4b840398b7d206e222ed31c [id=c96aada642b84c20894926b34fbea386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.820906 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c528a1a9ae0541ba9d4a80e38e743969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.821110 1018 tablet_service.cc:1467] Processing CreateTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 (DEFAULT_TABLE table=default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.821521 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d (DEFAULT_TABLE table=default.Table_d47b6907b4ec4c8bbd9b27d5a1dc6be5 [id=6af19c1506c9416d979c0d938080d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.821956 1025 tablet_service.cc:1467] Processing CreateTablet for tablet c40e5019321c497b814a281e0ab1a7f7 (DEFAULT_TABLE table=default.Table_47b199ac3a5945978c6af0a999fe64c1 [id=579828ccd3744a63b1212b6de38038ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.822273 1033 tablet_service.cc:1467] Processing CreateTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 (DEFAULT_TABLE table=default.Table_5d6390d21a0241e1bd37f6158126ff0e [id=7bd35efd3fad42dab4f6aa66d998c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.822571 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1589b79a56148b6ab2946943de39fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.825109 753 tablet_service.cc:1467] Processing CreateTablet for tablet c40e5019321c497b814a281e0ab1a7f7 (DEFAULT_TABLE table=default.Table_47b199ac3a5945978c6af0a999fe64c1 [id=579828ccd3744a63b1212b6de38038ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.826629 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 (DEFAULT_TABLE table=default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee [id=00e9e64762824d89b0a757e997078b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.826937 1032 tablet_service.cc:1467] Processing CreateTablet for tablet b0fda753bbc940f7834507345f67651c (DEFAULT_TABLE table=default.Table_e03d934cb25e4e96b3c9670d7698253a [id=f556fe2500bb47efba4056c36ca34823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.827219 1028 tablet_service.cc:1467] Processing CreateTablet for tablet da2d6dda69da4a59b055106c1041ec2b (DEFAULT_TABLE table=default.Table_c3dd2b1d3c0c4b20a2fe5f6533f750db [id=ce415eef7536445b99cab5e0769d359b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.827487 1034 tablet_service.cc:1467] Processing CreateTablet for tablet c528a1a9ae0541ba9d4a80e38e743969 (DEFAULT_TABLE table=default.Table_31efa1aed4b840398b7d206e222ed31c [id=c96aada642b84c20894926b34fbea386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.827847 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 (DEFAULT_TABLE table=default.Table_f580256f4281489b85b9f46fcd381c5f [id=a6f0a5fcb43d4509a188259adfdd0192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.828341 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 4c112bab5c5a42efab7582d83ca7eac1 (DEFAULT_TABLE table=default.Table_3565d86ee629490f99713f2b667ee3d2 [id=b7903ec3ae3b4d0f8ef509c4c05f11f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.829840 1176 tablet_bootstrap.cc:492] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:43.810988 756 tablet_service.cc:1467] Processing CreateTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d (DEFAULT_TABLE table=default.Table_d47b6907b4ec4c8bbd9b27d5a1dc6be5 [id=6af19c1506c9416d979c0d938080d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.830021 1176 ts_tablet_manager.cc:1397] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.017s
I20250109 20:52:43.830976 1176 raft_consensus.cc:357] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.831254 1176 raft_consensus.cc:383] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.831355 1176 raft_consensus.cc:738] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.831593 1176 consensus_queue.cc:260] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.827823 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ba73bbb08a4db0adfc9e72a84a22e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.832007 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1925a82eb711409ab2c5429c61edf9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.832137 1176 ts_tablet_manager.cc:1428] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:43.832414 1176 tablet_bootstrap.cc:492] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:43.832855 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c528a1a9ae0541ba9d4a80e38e743969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.834491 1176 tablet_bootstrap.cc:654] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.835177 1165 tablet_bootstrap.cc:492] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:43.836828 1165 tablet_bootstrap.cc:654] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.843444 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1044c212aada4df382d147c9a15ea4a7
I20250109 20:52:43.845198 899 tablet_service.cc:1467] Processing CreateTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 (DEFAULT_TABLE table=default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.845603 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ba73bbb08a4db0adfc9e72a84a22e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.846216 896 tablet_service.cc:1467] Processing CreateTablet for tablet c40e5019321c497b814a281e0ab1a7f7 (DEFAULT_TABLE table=default.Table_47b199ac3a5945978c6af0a999fe64c1 [id=579828ccd3744a63b1212b6de38038ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.846536 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40e5019321c497b814a281e0ab1a7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.849004 892 tablet_service.cc:1467] Processing CreateTablet for tablet b0fda753bbc940f7834507345f67651c (DEFAULT_TABLE table=default.Table_e03d934cb25e4e96b3c9670d7698253a [id=f556fe2500bb47efba4056c36ca34823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.849159 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4c112bab5c5a42efab7582d83ca7eac1 (DEFAULT_TABLE table=default.Table_3565d86ee629490f99713f2b667ee3d2 [id=b7903ec3ae3b4d0f8ef509c4c05f11f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.849327 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fda753bbc940f7834507345f67651c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.849679 898 tablet_service.cc:1467] Processing CreateTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d (DEFAULT_TABLE table=default.Table_d47b6907b4ec4c8bbd9b27d5a1dc6be5 [id=6af19c1506c9416d979c0d938080d827]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.850001 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779153d4174b4887b5dc0bb355cb4a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.852504 890 tablet_service.cc:1467] Processing CreateTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 (DEFAULT_TABLE table=default.Table_f580256f4281489b85b9f46fcd381c5f [id=a6f0a5fcb43d4509a188259adfdd0192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.852823 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d303cd42a2428fa17f29b48c4f64f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.852985 891 tablet_service.cc:1467] Processing CreateTablet for tablet da2d6dda69da4a59b055106c1041ec2b (DEFAULT_TABLE table=default.Table_c3dd2b1d3c0c4b20a2fe5f6533f750db [id=ce415eef7536445b99cab5e0769d359b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.853307 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2d6dda69da4a59b055106c1041ec2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.856267 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 6acadf60d566407aabc1a2395d1d235f from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.856585 897 tablet_service.cc:1467] Processing CreateTablet for tablet f1589b79a56148b6ab2946943de39fb2 (DEFAULT_TABLE table=default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.856913 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1589b79a56148b6ab2946943de39fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.858362 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044c212aada4df382d147c9a15ea4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.858588 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c38c8f0299e49ff8aff99b17b7be01b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.858963 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 (DEFAULT_TABLE table=default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee [id=00e9e64762824d89b0a757e997078b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.859275 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa23057a60a4a3eb7755e799bc3b911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.861416 887 tablet_service.cc:1467] Processing CreateTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 (DEFAULT_TABLE table=default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.861652 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1925a82eb711409ab2c5429c61edf9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.863907 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c112bab5c5a42efab7582d83ca7eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.865392 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 89fb6aeedc77452f99c46f03d6d8d089 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.869406 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044c212aada4df382d147c9a15ea4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.874125 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2d6dda69da4a59b055106c1041ec2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.875726 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779153d4174b4887b5dc0bb355cb4a9d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:43.877260 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1044c212aada4df382d147c9a15ea4a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1044c212aada4df382d147c9a15ea4a7 already in progress: creating tablet
I20250109 20:52:43.878049 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2a54393ff3264c52bfdeee2cb9046828
I20250109 20:52:43.881503 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6acadf60d566407aabc1a2395d1d235f
I20250109 20:52:43.881911 448 catalog_manager.cc:6596] ReplaceTablet: tablet 6acadf60d566407aabc1a2395d1d235f deleted and replaced by tablet f672cbd6dffc49dbb69992ae7db979ea
I20250109 20:52:43.885493 752 tablet_service.cc:1467] Processing CreateTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 (DEFAULT_TABLE table=default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.886054 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.886317 757 tablet_service.cc:1467] Processing CreateTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 (DEFAULT_TABLE table=default.Table_f580256f4281489b85b9f46fcd381c5f [id=a6f0a5fcb43d4509a188259adfdd0192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.886564 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4f020c5ba643aeb11f8e08d748be46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.886590 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acadf60d566407aabc1a2395d1d235f with delete_type TABLET_DATA_DELETED (replaced by tablet f672cbd6dffc49dbb69992ae7db979ea) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.887673 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b683408175de45c0bf7d7f7e14e15447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:43.888070 1160 tablet_replica.cc:331] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
W20250109 20:52:43.888139 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b683408175de45c0bf7d7f7e14e15447 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.889889 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40e5019321c497b814a281e0ab1a7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.890106 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acadf60d566407aabc1a2395d1d235f with delete_type TABLET_DATA_DELETED (replaced by tablet f672cbd6dffc49dbb69992ae7db979ea) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.889914 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bea28fb446e49669133e21b02c416ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.890362 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_35995197ab7d4324ae39cee214b8316f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.890748 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35995197ab7d4324ae39cee214b8316f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.900452 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4cc95165e1cf46caaef790cd356c2f23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:43.900885 1160 raft_consensus.cc:2238] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:43.901082 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cc95165e1cf46caaef790cd356c2f23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.901230 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:43.902531 1160 raft_consensus.cc:2267] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.903666 918 raft_consensus.cc:1270] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Refusing update from remote peer 5fcc647ed479491b88e5ffacf2927a1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:43.907255 1160 ts_tablet_manager.cc:1905] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.907470 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40e5019321c497b814a281e0ab1a7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.910329 778 raft_consensus.cc:1270] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Refusing update from remote peer 5fcc647ed479491b88e5ffacf2927a1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:43.914485 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.914774 1157 tablet_replica.cc:331] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:43.914968 1157 raft_consensus.cc:2238] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:43.915318 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d303cd42a2428fa17f29b48c4f64f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.916265 1157 raft_consensus.cc:2267] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:43.916672 1158 tablet_bootstrap.cc:492] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:43.916678 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acadf60d566407aabc1a2395d1d235f with delete_type TABLET_DATA_DELETED (replaced by tablet f672cbd6dffc49dbb69992ae7db979ea) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.919068 748 tablet_service.cc:1467] Processing CreateTablet for tablet f1589b79a56148b6ab2946943de39fb2 (DEFAULT_TABLE table=default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.922041 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2ee62027a3084959b04a5bb59395f3b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.922703 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ee62027a3084959b04a5bb59395f3b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.923727 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044c212aada4df382d147c9a15ea4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.924530 1157 ts_tablet_manager.cc:1905] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:43.924784 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ba73bbb08a4db0adfc9e72a84a22e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.925369 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1589b79a56148b6ab2946943de39fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.925210 749 tablet_service.cc:1467] Processing CreateTablet for tablet b0fda753bbc940f7834507345f67651c (DEFAULT_TABLE table=default.Table_e03d934cb25e4e96b3c9670d7698253a [id=f556fe2500bb47efba4056c36ca34823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.929530 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fda753bbc940f7834507345f67651c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.931623 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 825efab5ff7b492ca3c7e3fe11ece098 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.932526 1158 ts_tablet_manager.cc:1397] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.116s user 0.017s sys 0.008s
I20250109 20:52:43.934247 1158 raft_consensus.cc:357] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.936594 1160 ts_tablet_manager.cc:1918] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.936776 1160 log.cc:1198] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/6c38c8f0299e49ff8aff99b17b7be01b
I20250109 20:52:43.937283 1160 ts_tablet_manager.cc:1939] T 6c38c8f0299e49ff8aff99b17b7be01b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:43.938205 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_41c33f8ca7514962afbb30abb1e704da" } modify_external_catalogs: true
I20250109 20:52:43.938602 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 6c38c8f0299e49ff8aff99b17b7be01b (table default.Table_70760925d6df4f0c961c825337270e1a [id=d9fb05f1ceef49638bbe6db78d2d17c2]) successfully deleted
I20250109 20:52:43.939046 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_41c33f8ca7514962afbb30abb1e704da" } modify_external_catalogs: true
W20250109 20:52:43.939769 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6acadf60d566407aabc1a2395d1d235f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6acadf60d566407aabc1a2395d1d235f already in progress: creating tablet
W20250109 20:52:43.940097 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2a54393ff3264c52bfdeee2cb9046828 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a54393ff3264c52bfdeee2cb9046828 already in progress: creating tablet
I20250109 20:52:43.940964 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4529d87eadd74f31b511ff7c6b9b40e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.941675 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4529d87eadd74f31b511ff7c6b9b40e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.943492 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c3dd2b1d3c0c4b20a2fe5f6533f750db" } new_table_name: "default.Table_2ec8f8fde48b4f38881f6209ee4f4bac" modify_external_catalogs: true
I20250109 20:52:43.944851 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2acc983c759e437cbce5b1a3a5e88cfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.945389 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2acc983c759e437cbce5b1a3a5e88cfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.947425 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779153d4174b4887b5dc0bb355cb4a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.947362 1158 raft_consensus.cc:383] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.947790 1158 raft_consensus.cc:738] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.948386 1158 consensus_queue.cc:260] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.949177 1158 ts_tablet_manager.cc:1428] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250109 20:52:43.952474 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:43.956960 1158 tablet_bootstrap.cc:492] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:43.957511 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:43.964422 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fda753bbc940f7834507345f67651c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.966673 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d47b6907b4ec4c8bbd9b27d5a1dc6be5" } new_table_name: "default.Table_0f1b6adda4ab4df18b08fb9f430907e8" modify_external_catalogs: true
I20250109 20:52:43.968452 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_78db655e42e74268bbdff89f440ddc18" } new_table_name: "default.Table_8d84afcb90094c7fa27ee7102fb5088d" modify_external_catalogs: true
I20250109 20:52:43.967183 1187 consensus_queue.cc:1035] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20250109 20:52:43.970038 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f7166988e8c743c9896cbdbd048fa145"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.970463 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7166988e8c743c9896cbdbd048fa145 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.971709 1165 tablet_bootstrap.cc:492] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:43.971879 1165 ts_tablet_manager.cc:1397] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.137s user 0.039s sys 0.044s
I20250109 20:52:43.972007 1240 consensus_queue.cc:1035] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250109 20:52:43.972724 1165 raft_consensus.cc:357] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.972925 1165 raft_consensus.cc:383] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:43.972993 1165 raft_consensus.cc:738] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:43.973266 1165 consensus_queue.cc:260] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:43.973778 1165 ts_tablet_manager.cc:1428] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:43.974087 1165 tablet_bootstrap.cc:492] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:43.975695 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_846ea1135f4e4d599d2ec77d41525da4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:43.976877 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ad0a7364cd0d48caa4d583519597f8fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.977453 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad0a7364cd0d48caa4d583519597f8fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.978374 1166 ts_tablet_manager.cc:1918] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:43.978521 1166 log.cc:1198] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/d2aacfc7eea3490dbeb9d5c82d48766d
I20250109 20:52:43.979539 1165 tablet_bootstrap.cc:654] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:43.981058 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:43.981606 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:43.981029 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_846ea1135f4e4d599d2ec77d41525da4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:43.985168 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0463dd5de987418d8ea1b102b09137bf (DEFAULT_TABLE table=default.Table_7541b9bb11c44022ae972d09079fd75d [id=50837eb593734f66a03e29d8edbc63a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.986616 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 175bf6c335a8405c97b296cf4718610b from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:43.987159 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a4a01eeb3aa44fdc9926e35346c10db5" } new_table_name: "default.Table_0191c74c02514d34a53c6db0d807f298" modify_external_catalogs: true
I20250109 20:52:43.988965 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11d303cd42a2428fa17f29b48c4f64f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.991225 1034 tablet_service.cc:1467] Processing CreateTablet for tablet c643c1ca41e048fdab95a067f899342d (DEFAULT_TABLE table=default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:43.991478 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da2d6dda69da4a59b055106c1041ec2b consensus state has no leader
I20250109 20:52:43.992965 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d (DEFAULT_TABLE table=default.Table_41c33f8ca7514962afbb30abb1e704da [id=873b95c3a72b4e2f9d0e89183692ac52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.993264 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c112bab5c5a42efab7582d83ca7eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.993355 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.996969 899 tablet_service.cc:1467] Processing CreateTablet for tablet c643c1ca41e048fdab95a067f899342d (DEFAULT_TABLE table=default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:43.997354 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c643c1ca41e048fdab95a067f899342d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:43.999929 891 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.000491 1160 tablet_replica.cc:331] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:44.000671 1160 raft_consensus.cc:2238] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.000732 897 tablet_service.cc:1467] Processing CreateTablet for tablet 0463dd5de987418d8ea1b102b09137bf (DEFAULT_TABLE table=default.Table_7541b9bb11c44022ae972d09079fd75d [id=50837eb593734f66a03e29d8edbc63a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.000867 1160 raft_consensus.cc:2267] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.001093 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0463dd5de987418d8ea1b102b09137bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.001631 1160 ts_tablet_manager.cc:1905] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.002966 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d (DEFAULT_TABLE table=default.Table_41c33f8ca7514962afbb30abb1e704da [id=873b95c3a72b4e2f9d0e89183692ac52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.004860 746 tablet_service.cc:1467] Processing CreateTablet for tablet c643c1ca41e048fdab95a067f899342d (DEFAULT_TABLE table=default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.005836 747 tablet_service.cc:1467] Processing CreateTablet for tablet 0463dd5de987418d8ea1b102b09137bf (DEFAULT_TABLE table=default.Table_7541b9bb11c44022ae972d09079fd75d [id=50837eb593734f66a03e29d8edbc63a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.006181 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.006364 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c643c1ca41e048fdab95a067f899342d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.007738 1160 ts_tablet_manager.cc:1918] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.007872 1160 log.cc:1198] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f6a1a6a0a1f4411aa0b57481b6291808
I20250109 20:52:44.008235 1160 ts_tablet_manager.cc:1939] T f6a1a6a0a1f4411aa0b57481b6291808 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:44.008934 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f6a1a6a0a1f4411aa0b57481b6291808 (table default.Table_dab2393d93b14ec79d96807a37360e33 [id=fcec4978e3584a9cabd5641b5ffb7340]) successfully deleted
I20250109 20:52:44.009436 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0463dd5de987418d8ea1b102b09137bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.014806 1166 ts_tablet_manager.cc:1939] T d2aacfc7eea3490dbeb9d5c82d48766d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.015790 1166 tablet_replica.cc:331] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.016240 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet d2aacfc7eea3490dbeb9d5c82d48766d (table default.Table_8c0e3ac68686429dadf4fcc158cf285b [id=0acf665b047a4f1dac232ef98359b372]) successfully deleted
I20250109 20:52:44.016361 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa23057a60a4a3eb7755e799bc3b911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.019891 1166 raft_consensus.cc:2238] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.020084 1166 raft_consensus.cc:2267] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.020239 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c643c1ca41e048fdab95a067f899342d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.026712 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4cc95165e1cf46caaef790cd356c2f23" } modify_external_catalogs: true
I20250109 20:52:44.026916 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4cc95165e1cf46caaef790cd356c2f23" } modify_external_catalogs: true
I20250109 20:52:44.027855 1166 ts_tablet_manager.cc:1905] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:44.029320 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 779153d4174b4887b5dc0bb355cb4a9d consensus state has no leader
I20250109 20:52:44.030596 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cabbd075716c4d18aba39e549c7f0043" } modify_external_catalogs: true
I20250109 20:52:44.030820 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cabbd075716c4d18aba39e549c7f0043" } modify_external_catalogs: true
I20250109 20:52:44.031684 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4529d87eadd74f31b511ff7c6b9b40e3" } new_table_name: "default.Table_dc41b4efb2f245f29da9d86c1d4aef95" modify_external_catalogs: true
I20250109 20:52:44.037107 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0463dd5de987418d8ea1b102b09137bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:44.044775 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c7ed9e8c921491c9d911d662f500451 consensus state has no leader
W20250109 20:52:44.049750 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed4866f6007d411da5cf4832fa92ece9 consensus state has no leader
I20250109 20:52:44.050005 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d (DEFAULT_TABLE table=default.Table_41c33f8ca7514962afbb30abb1e704da [id=873b95c3a72b4e2f9d0e89183692ac52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.051237 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f9f28decdfb64455809f7b283552a4a5" } modify_external_catalogs: true
I20250109 20:52:44.051465 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f9f28decdfb64455809f7b283552a4a5" } modify_external_catalogs: true
I20250109 20:52:44.055691 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4e9cf5b77ccd4d65b023a9517efd2b0d
I20250109 20:52:44.057116 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.057468 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.057299 897 tablet_service.cc:1467] Processing CreateTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e (DEFAULT_TABLE table=default.Table_cabbd075716c4d18aba39e549c7f0043 [id=aa04f9d0face4e1b84edffbeea2b524b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:44.058374 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e9cf5b77ccd4d65b023a9517efd2b0d already in progress: creating tablet
I20250109 20:52:44.058461 899 tablet_service.cc:1467] Processing CreateTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d (DEFAULT_TABLE table=default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.058696 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e (DEFAULT_TABLE table=default.Table_cabbd075716c4d18aba39e549c7f0043 [id=aa04f9d0face4e1b84edffbeea2b524b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.059083 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8590e03e5d8e4f38aca6d4190a55771e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.058571 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8590e03e5d8e4f38aca6d4190a55771e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.062858 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08ddedf78c0442daa9b1d05334f12b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.063462 1158 tablet_bootstrap.cc:654] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.062029 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d (DEFAULT_TABLE table=default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.064348 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08ddedf78c0442daa9b1d05334f12b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.065225 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 7014756ffe2f4ea29b0ac7137c0f17eb
I20250109 20:52:44.067430 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 54d11b9148dc42bc91348e837af73972 (DEFAULT_TABLE table=default.Table_1e48be0e18b145708a9b6fa28ae21112 [id=194d1be4f3d946ec8c493ffcc5a23899]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.067896 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d11b9148dc42bc91348e837af73972. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.066983 891 tablet_service.cc:1467] Processing CreateTablet for tablet 54d11b9148dc42bc91348e837af73972 (DEFAULT_TABLE table=default.Table_1e48be0e18b145708a9b6fa28ae21112 [id=194d1be4f3d946ec8c493ffcc5a23899]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.069095 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d11b9148dc42bc91348e837af73972. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.070923 896 tablet_service.cc:1467] Processing CreateTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d (DEFAULT_TABLE table=default.Table_1838fc762deb4f71b0fd91a2eaa397ea [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.071331 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607a78e281594e9a92c6aa63ee7bdf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.073055 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 6a334724611e4303ba9d987033d872ba (DEFAULT_TABLE table=default.Table_c3c448aaf88a40de8c2d09f88e78442f [id=b1346293729744dda93f981b02aa3124]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.073488 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a334724611e4303ba9d987033d872ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.077028 898 tablet_service.cc:1467] Processing CreateTablet for tablet cba0ffb1b394431ba30918911cdee7b7 (DEFAULT_TABLE table=default.Table_609a0ae37cfd4b6d8939b4829beb84d6 [id=f06f85dff8684b1cac234aa2e35b6598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.077383 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba0ffb1b394431ba30918911cdee7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.085839 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d (DEFAULT_TABLE table=default.Table_1838fc762deb4f71b0fd91a2eaa397ea [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.086360 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607a78e281594e9a92c6aa63ee7bdf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.086390 1033 tablet_service.cc:1467] Processing CreateTablet for tablet cba0ffb1b394431ba30918911cdee7b7 (DEFAULT_TABLE table=default.Table_609a0ae37cfd4b6d8939b4829beb84d6 [id=f06f85dff8684b1cac234aa2e35b6598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.086823 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba0ffb1b394431ba30918911cdee7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.090495 893 tablet_service.cc:1467] Processing CreateTablet for tablet 6a334724611e4303ba9d987033d872ba (DEFAULT_TABLE table=default.Table_c3c448aaf88a40de8c2d09f88e78442f [id=b1346293729744dda93f981b02aa3124]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.090973 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a334724611e4303ba9d987033d872ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.100225 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.101962 1166 ts_tablet_manager.cc:1918] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.102087 1166 log.cc:1198] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/26f5b4af510942c8b50838df3bc8d8fc
I20250109 20:52:44.102355 1166 ts_tablet_manager.cc:1939] T 26f5b4af510942c8b50838df3bc8d8fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.107966 1165 tablet_bootstrap.cc:492] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.108106 1165 ts_tablet_manager.cc:1397] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.134s user 0.040s sys 0.031s
I20250109 20:52:44.108681 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 26f5b4af510942c8b50838df3bc8d8fc (table default.Table_3eef1bf244124c8d8d9bda7d0699a6f7 [id=0048f4157ceb4c098d924be9004eeaa1]) successfully deleted
I20250109 20:52:44.108934 1165 raft_consensus.cc:357] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.109134 1165 raft_consensus.cc:383] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.109218 1165 raft_consensus.cc:738] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.109428 1165 consensus_queue.cc:260] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.109994 1165 ts_tablet_manager.cc:1428] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:44.110304 1165 tablet_bootstrap.cc:492] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:44.110340 1176 tablet_bootstrap.cc:492] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
W20250109 20:52:44.111449 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:39231, user_credentials={real_user=slave}} blocked reactor thread for 48383.5us
I20250109 20:52:44.112217 1165 tablet_bootstrap.cc:654] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.117513 919 raft_consensus.cc:1270] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:44.119215 748 tablet_service.cc:1467] Processing CreateTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e (DEFAULT_TABLE table=default.Table_cabbd075716c4d18aba39e549c7f0043 [id=aa04f9d0face4e1b84edffbeea2b524b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.119742 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8590e03e5d8e4f38aca6d4190a55771e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.120350 447 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cdd067c335354357b44dac433f8c1523 (table default.Table_922405f1ef264d2a9457f48176a918a6 [id=8c860f8c96254692906f892a25530962]). Expected version 4 got 0
I20250109 20:52:44.121446 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_922405f1ef264d2a9457f48176a918a6" } modify_external_catalogs: true
I20250109 20:52:44.121668 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_922405f1ef264d2a9457f48176a918a6" } modify_external_catalogs: true
I20250109 20:52:44.123592 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_98434239d23e44ad848ad3a212ae9aac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.124182 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98434239d23e44ad848ad3a212ae9aac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.124222 1165 tablet_bootstrap.cc:492] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.124382 1165 ts_tablet_manager.cc:1397] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20250109 20:52:44.125351 1165 raft_consensus.cc:357] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.125360 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e182e0b0c2b6434eb2acce63b8935309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:44.125674 1165 raft_consensus.cc:383] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250109 20:52:44.125792 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e182e0b0c2b6434eb2acce63b8935309 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.125792 1165 raft_consensus.cc:738] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.126073 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8590e03e5d8e4f38aca6d4190a55771e
I20250109 20:52:44.126138 1165 consensus_queue.cc:260] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.126924 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a97de1805a3143398c7a97e52e5d2e64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.127487 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a97de1805a3143398c7a97e52e5d2e64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.127682 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:44.128383 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8590e03e5d8e4f38aca6d4190a55771e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8590e03e5d8e4f38aca6d4190a55771e already in progress: creating tablet
I20250109 20:52:44.128966 1165 ts_tablet_manager.cc:1428] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:44.130128 1165 tablet_bootstrap.cc:492] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:44.131770 1165 tablet_bootstrap.cc:654] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:44.133436 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7c277dcc04d4acdbc116e41ad41d7cf has no consensus state
I20250109 20:52:44.135247 758 tablet_service.cc:1467] Processing CreateTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d (DEFAULT_TABLE table=default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.135386 1176 ts_tablet_manager.cc:1397] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.278s user 0.026s sys 0.013s
I20250109 20:52:44.135788 1055 raft_consensus.cc:1270] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:44.135665 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08ddedf78c0442daa9b1d05334f12b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.136632 1166 tablet_replica.cc:331] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.140137 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.139376 1231 consensus_queue.cc:1035] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250109 20:52:44.140997 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.141328 1176 raft_consensus.cc:357] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.141649 1231 consensus_queue.cc:1035] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:44.148481 1176 raft_consensus.cc:383] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.148675 1176 raft_consensus.cc:738] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.147837 744 tablet_service.cc:1467] Processing CreateTablet for tablet 54d11b9148dc42bc91348e837af73972 (DEFAULT_TABLE table=default.Table_1e48be0e18b145708a9b6fa28ae21112 [id=194d1be4f3d946ec8c493ffcc5a23899]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.148109 742 tablet_service.cc:1467] Processing CreateTablet for tablet cba0ffb1b394431ba30918911cdee7b7 (DEFAULT_TABLE table=default.Table_609a0ae37cfd4b6d8939b4829beb84d6 [id=f06f85dff8684b1cac234aa2e35b6598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.149716 1176 consensus_queue.cc:260] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.150673 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.155548 1176 ts_tablet_manager.cc:1428] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20250109 20:52:44.159930 1176 tablet_bootstrap.cc:492] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.160741 1166 raft_consensus.cc:2238] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.161317 1166 raft_consensus.cc:2267] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.153551 743 tablet_service.cc:1467] Processing CreateTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d (DEFAULT_TABLE table=default.Table_1838fc762deb4f71b0fd91a2eaa397ea [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.156069 745 tablet_service.cc:1467] Processing CreateTablet for tablet 6a334724611e4303ba9d987033d872ba (DEFAULT_TABLE table=default.Table_c3c448aaf88a40de8c2d09f88e78442f [id=b1346293729744dda93f981b02aa3124]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.162189 1158 tablet_bootstrap.cc:492] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.162330 1158 ts_tablet_manager.cc:1397] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.209s user 0.004s sys 0.008s
I20250109 20:52:44.163033 1158 raft_consensus.cc:357] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.163229 1158 raft_consensus.cc:383] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.163318 1158 raft_consensus.cc:738] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.163553 1158 consensus_queue.cc:260] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.164080 1158 ts_tablet_manager.cc:1428] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:44.170388 1166 ts_tablet_manager.cc:1905] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.171125 1157 ts_tablet_manager.cc:1918] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.171285 1157 log.cc:1198] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/26f5b4af510942c8b50838df3bc8d8fc
I20250109 20:52:44.171701 1157 ts_tablet_manager.cc:1939] T 26f5b4af510942c8b50838df3bc8d8fc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:44.176311 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54d11b9148dc42bc91348e837af73972. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.177079 1176 tablet_bootstrap.cc:654] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:44.183537 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f7c277dcc04d4acdbc116e41ad41d7cf (table default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f7c277dcc04d4acdbc116e41ad41d7cf has no consensus state
I20250109 20:52:44.185484 1270 tablet.cc:1721] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:44.189652 1268 raft_consensus.cc:491] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.189847 1268 raft_consensus.cc:513] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.190878 1268 leader_election.cc:290] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:44.193113 1166 ts_tablet_manager.cc:1918] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.197301 1166 log.cc:1198] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/103722f5a2314bbeb9b141bb0a460ece
I20250109 20:52:44.197516 1158 tablet_bootstrap.cc:492] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:44.198608 1176 tablet_bootstrap.cc:492] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.199234 1158 tablet_bootstrap.cc:654] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.200872 1176 ts_tablet_manager.cc:1397] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.004s
I20250109 20:52:44.199445 1166 ts_tablet_manager.cc:1939] T 103722f5a2314bbeb9b141bb0a460ece P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.203284 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 103722f5a2314bbeb9b141bb0a460ece (table default.Table_0710fdbfdd7a4e7b892b501e77fc2e72 [id=606e0f04684243e8a30813edb6e96766]) successfully deleted
I20250109 20:52:44.204524 1176 raft_consensus.cc:357] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.206743 1176 raft_consensus.cc:383] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.207705 1157 tablet_replica.cc:331] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:44.205013 1166 tablet_replica.cc:331] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.207963 1157 raft_consensus.cc:2238] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.208118 1166 raft_consensus.cc:2238] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.208140 1157 raft_consensus.cc:2267] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.208524 1166 raft_consensus.cc:2267] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.209251 1157 ts_tablet_manager.cc:1905] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.209333 1187 raft_consensus.cc:491] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.209484 1187 raft_consensus.cc:513] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.210022 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 26f5b4af510942c8b50838df3bc8d8fc (table default.Table_3eef1bf244124c8d8d9bda7d0699a6f7 [id=0048f4157ceb4c098d924be9004eeaa1]) successfully deleted
I20250109 20:52:44.210181 1187 leader_election.cc:290] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:52:44.210417 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2a54393ff3264c52bfdeee2cb9046828 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a54393ff3264c52bfdeee2cb9046828 already in progress: creating tablet
W20250109 20:52:44.210703 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6acadf60d566407aabc1a2395d1d235f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6acadf60d566407aabc1a2395d1d235f already in progress: creating tablet
W20250109 20:52:44.210968 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1044c212aada4df382d147c9a15ea4a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1044c212aada4df382d147c9a15ea4a7 already in progress: creating tablet
I20250109 20:52:44.211097 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb6aeedc77452f99c46f03d6d8d089" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:44.211958 990 leader_election.cc:343] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.207711 1176 raft_consensus.cc:738] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.212419 1166 ts_tablet_manager.cc:1905] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.212918 1176 consensus_queue.cc:260] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.217250 1158 tablet_bootstrap.cc:492] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.217417 1158 ts_tablet_manager.cc:1397] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.008s
I20250109 20:52:44.217944 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba0ffb1b394431ba30918911cdee7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.218150 1158 raft_consensus.cc:357] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.218408 1158 raft_consensus.cc:383] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.218487 1158 raft_consensus.cc:738] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.218717 1158 consensus_queue.cc:260] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.219242 1158 ts_tablet_manager.cc:1428] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:44.219348 1176 ts_tablet_manager.cc:1428] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250109 20:52:44.219554 1158 tablet_bootstrap.cc:492] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:44.221357 1158 tablet_bootstrap.cc:654] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.221691 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a334724611e4303ba9d987033d872ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.222199 1176 tablet_bootstrap.cc:492] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.225167 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607a78e281594e9a92c6aa63ee7bdf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.228924 1176 tablet_bootstrap.cc:654] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.235958 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2aacfc7eea3490dbeb9d5c82d48766d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:44.239147 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_31efa1aed4b840398b7d206e222ed31c" } new_table_name: "default.Table_45d77a3e61d14f468ad3e271195a1cd9" modify_external_catalogs: true
I20250109 20:52:44.239281 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_65cb0c2ed23b450ca192c6f61becbc51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.239893 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65cb0c2ed23b450ca192c6f61becbc51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.240180 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2ee62027a3084959b04a5bb59395f3b0" } new_table_name: "default.Table_d8369c071899441a9b05f8910c9743c2" modify_external_catalogs: true
I20250109 20:52:44.241557 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4792b8689a12402f9946e87ad47dcbbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:44.241722 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e664b125305d4f36b75af160a7253520"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.242133 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4792b8689a12402f9946e87ad47dcbbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:44.242285 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e664b125305d4f36b75af160a7253520 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:44.243484 712 leader_election.cc:343] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.247118 916 raft_consensus.cc:3054] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:44.247794 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 with delete_type TABLET_DATA_DELETED (replaced by tablet 779153d4174b4887b5dc0bb355cb4a9d) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.257187 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 66c6016d88b54eb390f880945ddb1ea6
I20250109 20:52:44.258090 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cdd067c335354357b44dac433f8c1523
I20250109 20:52:44.259068 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.259083 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb6aeedc77452f99c46f03d6d8d089" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:44.259687 897 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd067c335354357b44dac433f8c1523 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:44.260303 989 leader_election.cc:343] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.260442 989 leader_election.cc:304] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:44.260378 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d59d31f7bc514ff38a99d27cbe8170f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:44.260815 1187 raft_consensus.cc:2743] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:44.261101 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 66c6016d88b54eb390f880945ddb1ea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66c6016d88b54eb390f880945ddb1ea6 already in progress: creating tablet
I20250109 20:52:44.261359 746 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd067c335354357b44dac433f8c1523 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.261317 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_846ea1135f4e4d599d2ec77d41525da4" } new_table_name: "default.Table_9d14aaae5e2d40a3842ad98fabf33015" modify_external_catalogs: true
I20250109 20:52:44.262475 1160 tablet_replica.cc:331] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:44.262529 1257 mvcc.cc:204] Tried to move back new op lower bound from 7112523628085932032 to 7112523626727432192. Current Snapshot: MvccSnapshot[applied={T|T < 7112523628085932032}]
I20250109 20:52:44.262921 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_77d18f6149124e54b0a321ccd3bd8fab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.263373 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77d18f6149124e54b0a321ccd3bd8fab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.263952 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:44.264083 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d59d31f7bc514ff38a99d27cbe8170f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.265952 1270 tablet.cc:1716] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Skipping requested alter to schema version 4, tablet already version 4
W20250109 20:52:44.265867 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 129488us
I20250109 20:52:44.266438 1275 tablet.cc:1721] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:44.266849 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_709427497f3a436a99bee6129d87efb6" } new_table_name: "default.Table_b7ff81513ee94c0bbc84bb3eeea7f144" modify_external_catalogs: true
I20250109 20:52:44.267196 1055 raft_consensus.cc:1270] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:44.269295 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd067c335354357b44dac433f8c1523 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.269289 1263 mvcc.cc:204] Tried to move back new op lower bound from 7112523628085932032 to 7112523626727432192. Current Snapshot: MvccSnapshot[applied={T|T < 7112523628085932032}]
I20250109 20:52:44.268131 1157 ts_tablet_manager.cc:1918] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.269575 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.269737 1157 log.cc:1198] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/d2aacfc7eea3490dbeb9d5c82d48766d
I20250109 20:52:44.269902 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet bd445a1adba34529aa3f7a44b991b177 with delete_type TABLET_DATA_DELETED (replaced by tablet 5496b2f965064f7788ebd74c62d1240a) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.270483 1157 ts_tablet_manager.cc:1939] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:44.272673 1231 consensus_queue.cc:1035] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:44.277412 1165 tablet_bootstrap.cc:492] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.278048 1165 ts_tablet_manager.cc:1397] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.148s user 0.030s sys 0.032s
I20250109 20:52:44.278626 1157 tablet_replica.cc:331] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:44.279249 1157 raft_consensus.cc:2238] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.279959 1157 raft_consensus.cc:2267] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.280617 1165 raft_consensus.cc:357] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.281039 1165 raft_consensus.cc:383] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.281232 1165 raft_consensus.cc:738] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.281677 1165 consensus_queue.cc:260] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.282847 1165 ts_tablet_manager.cc:1428] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:44.283246 1165 tablet_bootstrap.cc:492] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:52:44.285171 713 leader_election.cc:343] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: d2aacfc7eea3490dbeb9d5c82d48766d
I20250109 20:52:44.286029 713 leader_election.cc:304] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:44.286248 1165 tablet_bootstrap.cc:654] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:44.286520 713 raft_consensus.cc:2681] T d2aacfc7eea3490dbeb9d5c82d48766d P 09833ea46b7141d58c3899dace69e49f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:52:44.286756 1160 raft_consensus.cc:2238] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.287245 1160 pending_rounds.cc:62] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Trying to abort 2 pending ops.
I20250109 20:52:44.287420 1160 pending_rounds.cc:69] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112523628085932032 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "cdd067c335354357b44dac433f8c1523" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 4 new_table_name: "default.Table_922405f1ef264d2a9457f48176a918a6" dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" new_extra_config { } }
I20250109 20:52:44.287807 1166 ts_tablet_manager.cc:1918] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.288257 1160 pending_rounds.cc:69] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523626727432192 op_type: NO_OP noop_request { }
I20250109 20:52:44.288388 1166 log.cc:1198] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/e5d6dc79bf1e48ee9cf2bbb70463741d
I20250109 20:52:44.288422 1160 raft_consensus.cc:2883] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:44.288640 1160 raft_consensus.cc:2267] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.289080 1231 consensus_queue.cc:1035] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:44.288806 1166 ts_tablet_manager.cc:1939] T e5d6dc79bf1e48ee9cf2bbb70463741d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.293448 1176 tablet_bootstrap.cc:492] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.295531 1176 ts_tablet_manager.cc:1397] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.073s user 0.001s sys 0.004s
I20250109 20:52:44.296362 1165 tablet_bootstrap.cc:492] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.296377 1176 raft_consensus.cc:357] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.296615 1165 ts_tablet_manager.cc:1397] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20250109 20:52:44.296749 1176 raft_consensus.cc:383] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.296872 1176 raft_consensus.cc:738] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.297222 1176 consensus_queue.cc:260] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.297667 1165 raft_consensus.cc:357] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.297864 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet d2aacfc7eea3490dbeb9d5c82d48766d (table default.Table_8c0e3ac68686429dadf4fcc158cf285b [id=0acf665b047a4f1dac232ef98359b372]) successfully deleted
I20250109 20:52:44.298027 1165 raft_consensus.cc:383] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250109 20:52:44.298133 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e9cf5b77ccd4d65b023a9517efd2b0d already in progress: creating tablet
W20250109 20:52:44.298341 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8590e03e5d8e4f38aca6d4190a55771e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8590e03e5d8e4f38aca6d4190a55771e already in progress: creating tablet
I20250109 20:52:44.298807 1275 tablet.cc:1721] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:44.299068 1176 ts_tablet_manager.cc:1428] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:52:44.299607 1176 tablet_bootstrap.cc:492] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.298122 1165 raft_consensus.cc:738] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.300838 1165 consensus_queue.cc:260] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:44.302369 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1044c212aada4df382d147c9a15ea4a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1044c212aada4df382d147c9a15ea4a7 already in progress: creating tablet
I20250109 20:52:44.302424 1165 ts_tablet_manager.cc:1428] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250109 20:52:44.303000 1165 tablet_bootstrap.cc:492] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:44.306864 1165 tablet_bootstrap.cc:654] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.308570 1160 ts_tablet_manager.cc:1905] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.314062 1287 tablet.cc:1721] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250109 20:52:44.317157 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6acadf60d566407aabc1a2395d1d235f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6acadf60d566407aabc1a2395d1d235f already in progress: creating tablet
I20250109 20:52:44.317523 1166 tablet_replica.cc:331] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.317845 1166 raft_consensus.cc:2238] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.318080 1166 raft_consensus.cc:2267] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.318575 1166 ts_tablet_manager.cc:1905] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.318869 1176 tablet_bootstrap.cc:654] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.319296 1158 tablet_bootstrap.cc:492] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.319492 1158 ts_tablet_manager.cc:1397] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.100s user 0.000s sys 0.007s
I20250109 20:52:44.319967 1270 tablet.cc:1721] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:44.320441 1158 raft_consensus.cc:357] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.320742 1158 raft_consensus.cc:383] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.320886 1158 raft_consensus.cc:738] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.321192 1158 consensus_queue.cc:260] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.322757 1284 tablet.cc:1721] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:44.332938 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ea6be4d48d6041f493f1bf85432bfb0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.333499 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea6be4d48d6041f493f1bf85432bfb0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.340022 1270 tablet.cc:1721] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:44.341441 1160 ts_tablet_manager.cc:1918] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:44.341672 1160 log.cc:1198] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/cdd067c335354357b44dac433f8c1523
W20250109 20:52:44.342305 989 consensus_peers.cc:487] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:44.342355 1160 ts_tablet_manager.cc:1939] T cdd067c335354357b44dac433f8c1523 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:44.342715 1284 tablet.cc:1716] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Skipping requested alter to schema version 4, tablet already version 4
W20250109 20:52:44.350682 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffd6ce78438841daa663b6344f062019 has no consensus state
I20250109 20:52:44.351305 1285 tablet.cc:1721] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:44.352023 1284 tablet.cc:1716] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Skipping requested alter to schema version 3, tablet already version 3
I20250109 20:52:44.352552 1165 tablet_bootstrap.cc:492] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.352705 1165 ts_tablet_manager.cc:1397] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.011s
I20250109 20:52:44.353860 1270 tablet.cc:1716] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Skipping requested alter to schema version 3, tablet already version 3
I20250109 20:52:44.353915 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_2acf186aebe7415881f4304654dc6a83 [id=e4a51641b240493d9f69b9031ef5d4b1]). Expected version 3 got 0
I20250109 20:52:44.323771 1158 ts_tablet_manager.cc:1428] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250109 20:52:44.357086 1158 tablet_bootstrap.cc:492] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:44.358718 1158 tablet_bootstrap.cc:654] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.360620 1157 ts_tablet_manager.cc:1905] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.361984 1166 ts_tablet_manager.cc:1918] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.362080 1166 log.cc:1198] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6bea28fb446e49669133e21b02c416ad
I20250109 20:52:44.362154 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]). Expected version 1 got 0
I20250109 20:52:44.362386 1166 ts_tablet_manager.cc:1939] T 6bea28fb446e49669133e21b02c416ad P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.362730 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7" } modify_external_catalogs: true
I20250109 20:52:44.362906 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7" } modify_external_catalogs: true
I20250109 20:52:44.363250 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6bea28fb446e49669133e21b02c416ad (table default.Table_5ab3d2823bee4f739f414db53a1a6cd9 [id=06cecc207bb746dd9b0cb0f6eac4017e]) successfully deleted
I20250109 20:52:44.363413 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet cdd067c335354357b44dac433f8c1523 (table default.Table_922405f1ef264d2a9457f48176a918a6 [id=8c860f8c96254692906f892a25530962]) successfully deleted
I20250109 20:52:44.366880 1157 ts_tablet_manager.cc:1918] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250109 20:52:44.367025 1157 log.cc:1198] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/cdd067c335354357b44dac433f8c1523
I20250109 20:52:44.367456 1157 ts_tablet_manager.cc:1939] T cdd067c335354357b44dac433f8c1523 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:44.368414 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet cdd067c335354357b44dac433f8c1523 (table default.Table_922405f1ef264d2a9457f48176a918a6 [id=8c860f8c96254692906f892a25530962]) successfully deleted
W20250109 20:52:44.370082 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 66c6016d88b54eb390f880945ddb1ea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66c6016d88b54eb390f880945ddb1ea6 already in progress: creating tablet
I20250109 20:52:44.371392 1176 tablet_bootstrap.cc:492] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.371567 1176 ts_tablet_manager.cc:1397] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.008s
W20250109 20:52:44.372100 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c528a1a9ae0541ba9d4a80e38e743969 consensus state has no leader
I20250109 20:52:44.372277 1176 raft_consensus.cc:357] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.372447 1176 raft_consensus.cc:383] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.372510 1176 raft_consensus.cc:738] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:52:44.372614 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46a9910f88574eac90db9e00138c8dda has no consensus state
I20250109 20:52:44.372736 1176 consensus_queue.cc:260] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.372841 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 4f5a90e65e80494a8b451f2bc6baa2b1 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.375309 1176 ts_tablet_manager.cc:1428] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250109 20:52:44.377166 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 66c6016d88b54eb390f880945ddb1ea6 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.377511 1176 tablet_bootstrap.cc:492] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.379016 1176 tablet_bootstrap.cc:654] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.379281 744 tablet_service.cc:1467] Processing CreateTablet for tablet d08a3af516364157a07af7368a5f95f8 (DEFAULT_TABLE table=default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7 [id=ada5a5d915bf40e884cbd0e86a8e5c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.379714 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08a3af516364157a07af7368a5f95f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.379663 743 tablet_service.cc:1467] Processing CreateTablet for tablet 46a9910f88574eac90db9e00138c8dda (DEFAULT_TABLE table=default.Table_9d14aaae5e2d40a3842ad98fabf33015 [id=1e8fda0dc6034fdf9aa9b52712005046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.379846 1166 tablet_replica.cc:331] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.379997 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a9910f88574eac90db9e00138c8dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.380131 1166 raft_consensus.cc:2238] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.380236 748 tablet_service.cc:1467] Processing CreateTablet for tablet f672cbd6dffc49dbb69992ae7db979ea (DEFAULT_TABLE table=default.Table_08e4b27362104ad4af8816b4facabed7 [id=66c37d69f6bb4fe382c332f5c04e7a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.380366 1166 raft_consensus.cc:2267] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.380569 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f672cbd6dffc49dbb69992ae7db979ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.381301 1166 ts_tablet_manager.cc:1905] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.383898 746 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6ce78438841daa663b6344f062019 (DEFAULT_TABLE table=default.Table_d8369c071899441a9b05f8910c9743c2 [id=b939987c4a0f426b846403a8579622c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.384284 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd6ce78438841daa663b6344f062019. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.384428 753 tablet_service.cc:1467] Processing CreateTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 (DEFAULT_TABLE table=default.Table_2acc983c759e437cbce5b1a3a5e88cfe [id=b64c15100a8344a6bc35a896aec8270c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.384765 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d981c7f13c3409f994bfdf0275e0116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.387915 756 tablet_service.cc:1467] Processing CreateTablet for tablet ff4f256d59a2459e95be0f950a5a86eb (DEFAULT_TABLE table=default.Table_35995197ab7d4324ae39cee214b8316f [id=c6b6eaf39b88459490f459e9268f4980]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.388257 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4f256d59a2459e95be0f950a5a86eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.394181 899 tablet_service.cc:1467] Processing CreateTablet for tablet 46a9910f88574eac90db9e00138c8dda (DEFAULT_TABLE table=default.Table_9d14aaae5e2d40a3842ad98fabf33015 [id=1e8fda0dc6034fdf9aa9b52712005046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.394613 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a9910f88574eac90db9e00138c8dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.394551 897 tablet_service.cc:1467] Processing CreateTablet for tablet d08a3af516364157a07af7368a5f95f8 (DEFAULT_TABLE table=default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7 [id=ada5a5d915bf40e884cbd0e86a8e5c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.394862 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08a3af516364157a07af7368a5f95f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.396991 893 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6ce78438841daa663b6344f062019 (DEFAULT_TABLE table=default.Table_d8369c071899441a9b05f8910c9743c2 [id=b939987c4a0f426b846403a8579622c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.397284 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd6ce78438841daa663b6344f062019. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.400964 896 tablet_service.cc:1467] Processing CreateTablet for tablet ff4f256d59a2459e95be0f950a5a86eb (DEFAULT_TABLE table=default.Table_35995197ab7d4324ae39cee214b8316f [id=c6b6eaf39b88459490f459e9268f4980]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.401350 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4f256d59a2459e95be0f950a5a86eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.401841 1166 ts_tablet_manager.cc:1918] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.402081 1166 log.cc:1198] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/89fb6aeedc77452f99c46f03d6d8d089
I20250109 20:52:44.402936 1166 ts_tablet_manager.cc:1939] T 89fb6aeedc77452f99c46f03d6d8d089 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.403846 1166 tablet_replica.cc:331] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.404093 1166 raft_consensus.cc:2238] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:44.404793 1166 raft_consensus.cc:2267] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.405778 1166 ts_tablet_manager.cc:1905] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.405839 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 46a9910f88574eac90db9e00138c8dda (DEFAULT_TABLE table=default.Table_9d14aaae5e2d40a3842ad98fabf33015 [id=1e8fda0dc6034fdf9aa9b52712005046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.406163 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a9910f88574eac90db9e00138c8dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.408859 1176 tablet_bootstrap.cc:492] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.409039 1176 ts_tablet_manager.cc:1397] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.023s
I20250109 20:52:44.409103 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d08a3af516364157a07af7368a5f95f8 (DEFAULT_TABLE table=default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7 [id=ada5a5d915bf40e884cbd0e86a8e5c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.409552 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08a3af516364157a07af7368a5f95f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.409852 1176 raft_consensus.cc:357] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.410087 1176 raft_consensus.cc:383] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.410159 1176 raft_consensus.cc:738] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.410455 1176 consensus_queue.cc:260] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.411818 1166 ts_tablet_manager.cc:1918] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250109 20:52:44.412302 1176 ts_tablet_manager.cc:1428] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:44.412741 1176 tablet_bootstrap.cc:492] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.412902 1166 log.cc:1198] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/cdd067c335354357b44dac433f8c1523
I20250109 20:52:44.412981 1027 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6ce78438841daa663b6344f062019 (DEFAULT_TABLE table=default.Table_d8369c071899441a9b05f8910c9743c2 [id=b939987c4a0f426b846403a8579622c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.413355 1166 ts_tablet_manager.cc:1939] T cdd067c335354357b44dac433f8c1523 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.413466 1017 tablet_service.cc:1467] Processing CreateTablet for tablet ff4f256d59a2459e95be0f950a5a86eb (DEFAULT_TABLE table=default.Table_35995197ab7d4324ae39cee214b8316f [id=c6b6eaf39b88459490f459e9268f4980]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.413965 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f672cbd6dffc49dbb69992ae7db979ea (DEFAULT_TABLE table=default.Table_08e4b27362104ad4af8816b4facabed7 [id=66c37d69f6bb4fe382c332f5c04e7a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.414820 1176 tablet_bootstrap.cc:654] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.415156 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd6ce78438841daa663b6344f062019. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.416795 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41583
--webserver_interface=127.0.104.60
--webserver_port=46069
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:52:44.421597 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4f256d59a2459e95be0f950a5a86eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.422528 1176 tablet_bootstrap.cc:492] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.422706 1176 ts_tablet_manager.cc:1397] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250109 20:52:44.423559 1176 raft_consensus.cc:357] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.423803 1176 raft_consensus.cc:383] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.423935 1176 raft_consensus.cc:738] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.424227 1176 consensus_queue.cc:260] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.425060 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f672cbd6dffc49dbb69992ae7db979ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.425678 1176 ts_tablet_manager.cc:1428] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:44.426692 1176 tablet_bootstrap.cc:492] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.428149 1176 tablet_bootstrap.cc:654] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:44.432299 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2a54393ff3264c52bfdeee2cb9046828 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a54393ff3264c52bfdeee2cb9046828 already in progress: creating tablet
W20250109 20:52:44.432524 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e9cf5b77ccd4d65b023a9517efd2b0d already in progress: creating tablet
W20250109 20:52:44.432686 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8590e03e5d8e4f38aca6d4190a55771e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8590e03e5d8e4f38aca6d4190a55771e already in progress: creating tablet
I20250109 20:52:44.432930 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 89fb6aeedc77452f99c46f03d6d8d089 (table default.Table_0f1b6adda4ab4df18b08fb9f430907e8 [id=6af19c1506c9416d979c0d938080d827]) successfully deleted
I20250109 20:52:44.439522 744 tablet_service.cc:1467] Processing CreateTablet for tablet f0e8ee4b86a04199865acd5f69cda384 (DEFAULT_TABLE table=default.Table_b683408175de45c0bf7d7f7e14e15447 [id=2a713bcff6ae4df0a2a9ad75a6986394]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.439952 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e8ee4b86a04199865acd5f69cda384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.441488 898 tablet_service.cc:1467] Processing CreateTablet for tablet f672cbd6dffc49dbb69992ae7db979ea (DEFAULT_TABLE table=default.Table_08e4b27362104ad4af8816b4facabed7 [id=66c37d69f6bb4fe382c332f5c04e7a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.441874 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f672cbd6dffc49dbb69992ae7db979ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.448875 899 tablet_service.cc:1467] Processing CreateTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 (DEFAULT_TABLE table=default.Table_2acc983c759e437cbce5b1a3a5e88cfe [id=b64c15100a8344a6bc35a896aec8270c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.449343 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d981c7f13c3409f994bfdf0275e0116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.459288 746 tablet_service.cc:1467] Processing CreateTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf (DEFAULT_TABLE table=default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.459735 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c277dcc04d4acdbc116e41ad41d7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.460832 891 tablet_service.cc:1467] Processing CreateTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf (DEFAULT_TABLE table=default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.461205 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c277dcc04d4acdbc116e41ad41d7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.465044 756 tablet_service.cc:1467] Processing CreateTablet for tablet 6716e7428ce240fca7ca5b02f564a5df (DEFAULT_TABLE table=default.Table_f7166988e8c743c9896cbdbd048fa145 [id=49d5df19d5124b71993545f365062601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.465484 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6716e7428ce240fca7ca5b02f564a5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.466192 892 tablet_service.cc:1467] Processing CreateTablet for tablet 6716e7428ce240fca7ca5b02f564a5df (DEFAULT_TABLE table=default.Table_f7166988e8c743c9896cbdbd048fa145 [id=49d5df19d5124b71993545f365062601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.466521 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6716e7428ce240fca7ca5b02f564a5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.468999 743 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.469353 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9feb9de9dc4d22bafc0f7707943113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.473237 1240 raft_consensus.cc:491] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.369155 1165 raft_consensus.cc:357] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.475626 1166 tablet_replica.cc:331] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.476964 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 (DEFAULT_TABLE table=default.Table_2acc983c759e437cbce5b1a3a5e88cfe [id=b64c15100a8344a6bc35a896aec8270c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.477006 896 tablet_service.cc:1467] Processing CreateTablet for tablet f0e8ee4b86a04199865acd5f69cda384 (DEFAULT_TABLE table=default.Table_b683408175de45c0bf7d7f7e14e15447 [id=2a713bcff6ae4df0a2a9ad75a6986394]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.477326 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e8ee4b86a04199865acd5f69cda384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.477551 1240 raft_consensus.cc:513] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.478137 1240 leader_election.cc:290] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:44.478453 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cfa3005fdd40709a630d5bb80564db" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:44.478636 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cfa3005fdd40709a630d5bb80564db" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:44.479297 1018 tablet_service.cc:1467] Processing CreateTablet for tablet f0e8ee4b86a04199865acd5f69cda384 (DEFAULT_TABLE table=default.Table_b683408175de45c0bf7d7f7e14e15447 [id=2a713bcff6ae4df0a2a9ad75a6986394]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.479632 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet cdd067c335354357b44dac433f8c1523 (table default.Table_922405f1ef264d2a9457f48176a918a6 [id=8c860f8c96254692906f892a25530962]) successfully deleted
W20250109 20:52:44.479909 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 66c6016d88b54eb390f880945ddb1ea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66c6016d88b54eb390f880945ddb1ea6 already in progress: creating tablet
I20250109 20:52:44.480355 1165 raft_consensus.cc:383] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.481504 897 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.487434 1032 tablet_service.cc:1467] Processing CreateTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf (DEFAULT_TABLE table=default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.487942 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.489425 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9feb9de9dc4d22bafc0f7707943113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.481280 1165 raft_consensus.cc:738] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.491365 447 master_service.cc:946] ReplaceTablet: received request to replace tablet cdd067c335354357b44dac433f8c1523 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.492815 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d981c7f13c3409f994bfdf0275e0116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.493009 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c277dcc04d4acdbc116e41ad41d7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.493216 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b9feb9de9dc4d22bafc0f7707943113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.492329 1165 consensus_queue.cc:260] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.494482 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 6716e7428ce240fca7ca5b02f564a5df (DEFAULT_TABLE table=default.Table_f7166988e8c743c9896cbdbd048fa145 [id=49d5df19d5124b71993545f365062601]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:44.494824 989 leader_election.cc:343] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:44.495079 990 leader_election.cc:343] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.495445 990 leader_election.cc:304] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:44.495803 1166 raft_consensus.cc:2238] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.496330 1158 tablet_bootstrap.cc:492] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.496538 1158 ts_tablet_manager.cc:1397] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.140s user 0.008s sys 0.034s
I20250109 20:52:44.496606 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e8ee4b86a04199865acd5f69cda384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.497157 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6716e7428ce240fca7ca5b02f564a5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.497464 1158 raft_consensus.cc:357] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.497696 1158 raft_consensus.cc:383] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.497784 1158 raft_consensus.cc:738] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.498055 1158 consensus_queue.cc:260] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.498625 1158 ts_tablet_manager.cc:1428] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:44.499019 1158 tablet_bootstrap.cc:492] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:44.500562 1158 tablet_bootstrap.cc:654] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.501001 1165 ts_tablet_manager.cc:1428] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.148s user 0.002s sys 0.000s
I20250109 20:52:44.501320 1165 tablet_bootstrap.cc:492] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:44.501591 1166 raft_consensus.cc:2267] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.502290 1240 raft_consensus.cc:2743] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:44.503955 1165 tablet_bootstrap.cc:654] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.513572 1166 ts_tablet_manager.cc:1905] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.530256 1287 tablet.cc:1716] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Skipping requested alter to schema version 3, tablet already version 3
I20250109 20:52:44.530937 1158 tablet_bootstrap.cc:492] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.531113 1158 ts_tablet_manager.cc:1397] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.007s
I20250109 20:52:44.532042 1158 raft_consensus.cc:357] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.532249 1158 raft_consensus.cc:383] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.532343 1158 raft_consensus.cc:738] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.532577 1158 consensus_queue.cc:260] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.533083 1158 ts_tablet_manager.cc:1428] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:44.533411 1158 tablet_bootstrap.cc:492] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:44.535904 1165 tablet_bootstrap.cc:492] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.536091 1165 ts_tablet_manager.cc:1397] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.012s
I20250109 20:52:44.536353 1191 raft_consensus.cc:491] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.536501 1191 raft_consensus.cc:513] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.537158 1191 leader_election.cc:290] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:44.536980 1165 raft_consensus.cc:357] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.537662 1165 raft_consensus.cc:383] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.537779 1165 raft_consensus.cc:738] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:52:44.537891 854 leader_election.cc:343] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: e5d6dc79bf1e48ee9cf2bbb70463741d
I20250109 20:52:44.538035 1165 consensus_queue.cc:260] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.538529 1165 ts_tablet_manager.cc:1428] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:44.538846 1165 tablet_bootstrap.cc:492] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:44.540542 1165 tablet_bootstrap.cc:654] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.546536 1158 tablet_bootstrap.cc:654] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:44.546659 854 leader_election.cc:343] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: e5d6dc79bf1e48ee9cf2bbb70463741d
I20250109 20:52:44.546782 854 leader_election.cc:304] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:44.547165 1191 raft_consensus.cc:2743] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:44.548684 1240 raft_consensus.cc:491] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.548866 1240 raft_consensus.cc:513] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.549389 1240 leader_election.cc:290] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:44.549964 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeedf2e359c94a96ba6a518cefbb1fbe" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:44.550233 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeedf2e359c94a96ba6a518cefbb1fbe" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:44.550454 989 leader_election.cc:343] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.550469 778 raft_consensus.cc:2463] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:44.550814 990 leader_election.cc:304] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:44.551390 1240 raft_consensus.cc:2798] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:44.551489 1240 raft_consensus.cc:491] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:44.551589 1240 raft_consensus.cc:3054] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:44.553998 1240 raft_consensus.cc:513] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.554760 1240 leader_election.cc:290] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:44.554960 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeedf2e359c94a96ba6a518cefbb1fbe" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:44.554953 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeedf2e359c94a96ba6a518cefbb1fbe" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:44.555176 778 raft_consensus.cc:3054] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:52:44.555555 989 leader_election.cc:343] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.558210 778 raft_consensus.cc:2463] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:44.558550 990 leader_election.cc:304] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:44.558969 1240 raft_consensus.cc:2798] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:44.559123 1240 raft_consensus.cc:695] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:44.559376 1240 consensus_queue.cc:237] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.560165 1158 tablet_bootstrap.cc:492] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.560317 1158 ts_tablet_manager.cc:1397] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20250109 20:52:44.561100 1158 raft_consensus.cc:357] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.561300 1158 raft_consensus.cc:383] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.561378 1158 raft_consensus.cc:738] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.561610 1158 consensus_queue.cc:260] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.562168 1158 ts_tablet_manager.cc:1428] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:44.562434 1158 tablet_bootstrap.cc:492] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:44.568023 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 175bf6c335a8405c97b296cf4718610b from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.568430 455 master_service.cc:946] ReplaceTablet: received request to replace tablet b0fda753bbc940f7834507345f67651c from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.570009 1166 ts_tablet_manager.cc:1918] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:44.570291 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 660e60f368cb4e25b12ab3d7b2e314b1 consensus state has no leader
I20250109 20:52:44.570676 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_2acf186aebe7415881f4304654dc6a83 [id=e4a51641b240493d9f69b9031ef5d4b1] alter complete (version 3)
I20250109 20:52:44.571965 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3565d86ee629490f99713f2b667ee3d2" } new_table_name: "default.Table_8730181edd7f48769b95dd508118d1df" modify_external_catalogs: true
I20250109 20:52:44.573107 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a5bb2f6d26e3423e93672a1954c238a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.573698 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5bb2f6d26e3423e93672a1954c238a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.573841 1166 log.cc:1198] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/bd445a1adba34529aa3f7a44b991b177
I20250109 20:52:44.574240 1166 ts_tablet_manager.cc:1939] T bd445a1adba34529aa3f7a44b991b177 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.575584 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7d5333c1118343aeb596d21852bbedbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.576016 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d5333c1118343aeb596d21852bbedbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.577634 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e4f018070aac4f4298a611b89b0e5f6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.578030 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4f018070aac4f4298a611b89b0e5f6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.579167 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_51da7d20722144bf824bdc0388d68808" } modify_external_catalogs: true
I20250109 20:52:44.579339 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_51da7d20722144bf824bdc0388d68808" } modify_external_catalogs: true
I20250109 20:52:44.579468 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c3c448aaf88a40de8c2d09f88e78442f" } modify_external_catalogs: true
I20250109 20:52:44.579632 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c3c448aaf88a40de8c2d09f88e78442f" } modify_external_catalogs: true
I20250109 20:52:44.579972 1176 tablet_bootstrap.cc:492] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.580137 1176 ts_tablet_manager.cc:1397] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.153s user 0.004s sys 0.004s
I20250109 20:52:44.580920 1176 raft_consensus.cc:357] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.581177 1176 raft_consensus.cc:383] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.581261 1176 raft_consensus.cc:738] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.581490 1176 consensus_queue.cc:260] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.581929 1176 ts_tablet_manager.cc:1428] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:44.582299 1176 tablet_bootstrap.cc:492] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.582355 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet bd445a1adba34529aa3f7a44b991b177 (table default.Table_9af6ca0c3f0042358b13f0891ced6dea [id=16fc7304546a423abcbcd230d06efd59]) successfully deleted
I20250109 20:52:44.584395 1176 tablet_bootstrap.cc:654] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.584724 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d08a3af516364157a07af7368a5f95f8
I20250109 20:52:44.585995 896 tablet_service.cc:1514] Processing DeleteTablet for tablet d08a3af516364157a07af7368a5f95f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:44.586647 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d08a3af516364157a07af7368a5f95f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d08a3af516364157a07af7368a5f95f8 already in progress: creating tablet
I20250109 20:52:44.587414 746 tablet_service.cc:1514] Processing DeleteTablet for tablet d08a3af516364157a07af7368a5f95f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:44.588120 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d08a3af516364157a07af7368a5f95f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d08a3af516364157a07af7368a5f95f8 already in progress: creating tablet
I20250109 20:52:44.588441 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet d08a3af516364157a07af7368a5f95f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:44.589107 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d08a3af516364157a07af7368a5f95f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d08a3af516364157a07af7368a5f95f8 already in progress: creating tablet
I20250109 20:52:44.589396 1158 tablet_bootstrap.cc:654] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.590049 1165 tablet_bootstrap.cc:492] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.591334 1165 ts_tablet_manager.cc:1397] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.007s
I20250109 20:52:44.592236 1165 raft_consensus.cc:357] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.593899 1165 raft_consensus.cc:383] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.594247 1165 raft_consensus.cc:738] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.600251 1176 tablet_bootstrap.cc:492] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.600379 1176 ts_tablet_manager.cc:1397] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250109 20:52:44.601056 1176 raft_consensus.cc:357] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.601213 1176 raft_consensus.cc:383] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.601279 1176 raft_consensus.cc:738] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.601346 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_384c816ede6b4c558524a02495bb3dc0" } modify_external_catalogs: true
I20250109 20:52:44.601455 1176 consensus_queue.cc:260] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.601524 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9016eedbbe184e968d3a7d60d2286789" } modify_external_catalogs: true
I20250109 20:52:44.601503 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_384c816ede6b4c558524a02495bb3dc0" } modify_external_catalogs: true
I20250109 20:52:44.601649 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9016eedbbe184e968d3a7d60d2286789" } modify_external_catalogs: true
W20250109 20:52:44.602197 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c112bab5c5a42efab7582d83ca7eac1 consensus state has no leader
I20250109 20:52:44.602180 1176 ts_tablet_manager.cc:1428] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:44.602464 1176 tablet_bootstrap.cc:492] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.602541 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_834dbde1ccad4d8bb60efe0ae6ecf469"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.602978 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_834dbde1ccad4d8bb60efe0ae6ecf469 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.603680 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b0fda753bbc940f7834507345f67651c
I20250109 20:52:44.604175 455 catalog_manager.cc:6596] ReplaceTablet: tablet b0fda753bbc940f7834507345f67651c deleted and replaced by tablet cba68160be6145ddbbd2019c9b5f0076
I20250109 20:52:44.604430 746 tablet_service.cc:1514] Processing DeleteTablet for tablet b0fda753bbc940f7834507345f67651c with delete_type TABLET_DATA_DELETED (replaced by tablet cba68160be6145ddbbd2019c9b5f0076) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.604526 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_655ff82c5466442c818f93c7d408c157"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.604945 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_655ff82c5466442c818f93c7d408c157 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:44.605178 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b0fda753bbc940f7834507345f67651c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0fda753bbc940f7834507345f67651c already in progress: creating tablet
I20250109 20:52:44.605546 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b0fda753bbc940f7834507345f67651c with delete_type TABLET_DATA_DELETED (replaced by tablet cba68160be6145ddbbd2019c9b5f0076) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.605609 1176 tablet_bootstrap.cc:654] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:44.606312 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b0fda753bbc940f7834507345f67651c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0fda753bbc940f7834507345f67651c already in progress: creating tablet
I20250109 20:52:44.615459 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6a334724611e4303ba9d987033d872ba
I20250109 20:52:44.616117 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e8ceeee244274a419a60eb82212c9d81" } new_table_name: "default.Table_ab5a0c56116047958eb48524522f7b27" modify_external_catalogs: true
I20250109 20:52:44.616469 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7609a405eec042e4982438d13650f8a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.616950 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7609a405eec042e4982438d13650f8a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.618475 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_18addc41a8074f018e26f238d2e7ca70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.619545 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18addc41a8074f018e26f238d2e7ca70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.622331 1176 tablet_bootstrap.cc:492] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.622917 1176 ts_tablet_manager.cc:1397] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.003s
I20250109 20:52:44.623183 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a334724611e4303ba9d987033d872ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.623519 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a334724611e4303ba9d987033d872ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.623692 1176 raft_consensus.cc:357] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.623901 1176 raft_consensus.cc:383] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.623965 1176 raft_consensus.cc:738] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.624235 1176 consensus_queue.cc:260] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.624805 1273 raft_consensus.cc:491] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:52:44.624883 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6a334724611e4303ba9d987033d872ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a334724611e4303ba9d987033d872ba already in progress: creating tablet
I20250109 20:52:44.624962 1273 raft_consensus.cc:513] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
W20250109 20:52:44.625243 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6a334724611e4303ba9d987033d872ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a334724611e4303ba9d987033d872ba already in progress: creating tablet
I20250109 20:52:44.625360 1176 ts_tablet_manager.cc:1428] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:44.625672 1176 tablet_bootstrap.cc:492] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.625680 1273 leader_election.cc:290] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:44.627384 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cfa3005fdd40709a630d5bb80564db" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:44.627509 1176 tablet_bootstrap.cc:654] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.627583 1055 raft_consensus.cc:2463] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:44.628296 713 leader_election.cc:304] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:44.628605 1273 raft_consensus.cc:2798] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:44.628700 1273 raft_consensus.cc:491] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:44.628824 1273 raft_consensus.cc:3054] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:44.631294 1273 raft_consensus.cc:513] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.632314 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cfa3005fdd40709a630d5bb80564db" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:44.632486 1055 raft_consensus.cc:3054] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:44.633055 1165 consensus_queue.cc:260] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.605480 896 tablet_service.cc:1514] Processing DeleteTablet for tablet b0fda753bbc940f7834507345f67651c with delete_type TABLET_DATA_DELETED (replaced by tablet cba68160be6145ddbbd2019c9b5f0076) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:44.633855 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab593a602d61495eb7b323903ae610b6 consensus state has no leader
I20250109 20:52:44.623792 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a334724611e4303ba9d987033d872ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:44.634865 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b0fda753bbc940f7834507345f67651c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0fda753bbc940f7834507345f67651c already in progress: creating tablet
W20250109 20:52:44.635094 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6a334724611e4303ba9d987033d872ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a334724611e4303ba9d987033d872ba already in progress: creating tablet
I20250109 20:52:44.626653 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cfa3005fdd40709a630d5bb80564db" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:44.635347 1055 raft_consensus.cc:2463] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:44.635491 1165 ts_tablet_manager.cc:1428] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20250109 20:52:44.636620 1165 tablet_bootstrap.cc:492] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:52:44.636567 712 leader_election.cc:343] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.637249 713 leader_election.cc:304] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:44.637542 1268 raft_consensus.cc:2798] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:44.637691 1268 raft_consensus.cc:695] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:44.637974 1268 consensus_queue.cc:237] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.638695 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ea6be4d48d6041f493f1bf85432bfb0e" } new_table_name: "default.Table_cb4c99bfeae9477485df9f52b78ea759" modify_external_catalogs: true
I20250109 20:52:44.639065 1165 tablet_bootstrap.cc:654] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.640403 1191 raft_consensus.cc:491] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.640549 1191 raft_consensus.cc:513] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.640676 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c2a802cfb927494f92e603090370e543"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:44.641088 1191 leader_election.cc:290] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:44.641147 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2a802cfb927494f92e603090370e543 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.641198 1158 tablet_bootstrap.cc:492] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.641317 1158 ts_tablet_manager.cc:1397] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.000s
I20250109 20:52:44.642197 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b7ff81513ee94c0bbc84bb3eeea7f144" } new_table_name: "default.Table_ff7c9fe449d540f19d0ce42d44ba0b73" modify_external_catalogs: true
W20250109 20:52:44.642519 854 leader_election.cc:343] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: ba365ba04f6848328a9b14dc4c92ac2d
I20250109 20:52:44.642724 1158 raft_consensus.cc:357] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.642890 1158 raft_consensus.cc:383] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.642983 1158 raft_consensus.cc:738] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:44.643330 854 leader_election.cc:343] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: ba365ba04f6848328a9b14dc4c92ac2d
I20250109 20:52:44.643436 854 leader_election.cc:304] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:44.643873 1191 raft_consensus.cc:2743] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:44.643884 1158 consensus_queue.cc:260] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.644492 1158 ts_tablet_manager.cc:1428] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:44.644788 1158 tablet_bootstrap.cc:492] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:44.646544 1158 tablet_bootstrap.cc:654] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.649250 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cfa3005fdd40709a630d5bb80564db" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:44.649621 1273 leader_election.cc:290] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:44.649942 712 leader_election.cc:343] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.654491 1158 tablet_bootstrap.cc:492] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.654645 1158 ts_tablet_manager.cc:1397] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20250109 20:52:44.655489 1158 raft_consensus.cc:357] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.656173 1158 raft_consensus.cc:383] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.656395 1158 raft_consensus.cc:738] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.656878 1158 consensus_queue.cc:260] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.657868 1158 ts_tablet_manager.cc:1428] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250109 20:52:44.658130 1158 tablet_bootstrap.cc:492] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:44.658262 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 660e60f368cb4e25b12ab3d7b2e314b1 (table default.Table_b7ff81513ee94c0bbc84bb3eeea7f144 [id=028851fe80724d3cb4402101784bd12a]). Expected version 1 got 0
I20250109 20:52:44.659861 746 tablet_service.cc:1467] Processing CreateTablet for tablet d6b06e8616df461d89c58d414fc0cfbd (DEFAULT_TABLE table=default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.660233 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b06e8616df461d89c58d414fc0cfbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.660218 1034 tablet_service.cc:1467] Processing CreateTablet for tablet d6b06e8616df461d89c58d414fc0cfbd (DEFAULT_TABLE table=default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.660815 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b06e8616df461d89c58d414fc0cfbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.662199 1158 tablet_bootstrap.cc:654] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.665020 756 tablet_service.cc:1467] Processing CreateTablet for tablet 41de7acfa7f941caa94bdbb888cf841e (DEFAULT_TABLE table=default.Table_65cb0c2ed23b450ca192c6f61becbc51 [id=2504d7b1b64f477a8b7763d5d0a9ec6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.665045 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 41de7acfa7f941caa94bdbb888cf841e (DEFAULT_TABLE table=default.Table_65cb0c2ed23b450ca192c6f61becbc51 [id=2504d7b1b64f477a8b7763d5d0a9ec6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.665381 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41de7acfa7f941caa94bdbb888cf841e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.665493 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41de7acfa7f941caa94bdbb888cf841e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.666116 743 tablet_service.cc:1467] Processing CreateTablet for tablet 5e90cd406e394166b3e8dedebf7fd0fc (DEFAULT_TABLE table=default.Table_a97de1805a3143398c7a97e52e5d2e64 [id=e39eeed4512c4637b813b7ee38d4318d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.666486 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e90cd406e394166b3e8dedebf7fd0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.667716 1231 raft_consensus.cc:491] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.667893 1231 raft_consensus.cc:513] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.668649 1231 leader_election.cc:290] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:44.669530 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb] alter complete (version 1)
I20250109 20:52:44.669960 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb6aeedc77452f99c46f03d6d8d089" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:44.670574 712 leader_election.cc:343] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:44.671243 713 leader_election.cc:343] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 89fb6aeedc77452f99c46f03d6d8d089
I20250109 20:52:44.671392 713 leader_election.cc:304] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:44.675343 891 tablet_service.cc:1467] Processing CreateTablet for tablet 5e90cd406e394166b3e8dedebf7fd0fc (DEFAULT_TABLE table=default.Table_a97de1805a3143398c7a97e52e5d2e64 [id=e39eeed4512c4637b813b7ee38d4318d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.675837 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e90cd406e394166b3e8dedebf7fd0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.675918 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 58817dddbe8749249a96c60dee41b520 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.676146 1160 tablet_replica.cc:331] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:44.676365 1160 raft_consensus.cc:2238] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.676597 1160 raft_consensus.cc:2267] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.676865 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f5b4af510942c8b50838df3bc8d8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.677146 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ae6c4e835048bd8deed5ac9c4ae07a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.677295 1160 ts_tablet_manager.cc:1905] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.677366 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d6dc79bf1e48ee9cf2bbb70463741d with delete_type TABLET_DATA_DELETED (replaced by tablet 66c6016d88b54eb390f880945ddb1ea6) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.677863 897 tablet_service.cc:1467] Processing CreateTablet for tablet d6b06e8616df461d89c58d414fc0cfbd (DEFAULT_TABLE table=default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.678259 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b06e8616df461d89c58d414fc0cfbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.681015 896 tablet_service.cc:1467] Processing CreateTablet for tablet 41de7acfa7f941caa94bdbb888cf841e (DEFAULT_TABLE table=default.Table_65cb0c2ed23b450ca192c6f61becbc51 [id=2504d7b1b64f477a8b7763d5d0a9ec6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.681401 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41de7acfa7f941caa94bdbb888cf841e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.681974 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bea28fb446e49669133e21b02c416ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.682210 899 tablet_service.cc:1514] Processing DeleteTablet for tablet d2aacfc7eea3490dbeb9d5c82d48766d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.682391 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ba365ba04f6848328a9b14dc4c92ac2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.683406 1231 raft_consensus.cc:2743] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:44.685967 753 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd067c335354357b44dac433f8c1523 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.686203 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a1a6a0a1f4411aa0b57481b6291808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.686450 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 with delete_type TABLET_DATA_DELETED (replaced by tablet 779153d4174b4887b5dc0bb355cb4a9d) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.687299 745 tablet_service.cc:1514] Processing DeleteTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.687692 1160 ts_tablet_manager.cc:1918] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.687719 1157 tablet_replica.cc:331] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:44.687831 1160 log.cc:1198] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/58817dddbe8749249a96c60dee41b520
I20250109 20:52:44.687947 1157 raft_consensus.cc:2238] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.688123 1157 raft_consensus.cc:2267] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.688237 1160 ts_tablet_manager.cc:1939] T 58817dddbe8749249a96c60dee41b520 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:44.688848 1157 ts_tablet_manager.cc:1905] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:44.695693 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cdd067c335354357b44dac433f8c1523 because the tablet was not found. No further retry: Not found: Tablet not found: cdd067c335354357b44dac433f8c1523
I20250109 20:52:44.705156 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b646ab4659864a2d9f3707100706389a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.705936 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b646ab4659864a2d9f3707100706389a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.707160 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 58817dddbe8749249a96c60dee41b520 (table default.Table_d0a7a37106ea46e683ead0b81aa0585e [id=fc46d1a61d7d4769bfcf249712d2e2b9]) successfully deleted
I20250109 20:52:44.708031 1160 tablet_replica.cc:331] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:44.708570 1160 raft_consensus.cc:2238] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.709079 1160 raft_consensus.cc:2267] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.710112 1160 ts_tablet_manager.cc:1905] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.714004 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3000b819a4cc4960909b98febd400631 (DEFAULT_TABLE table=default.Table_77d18f6149124e54b0a321ccd3bd8fab [id=2d996a757dae433a9853245f2ceb1f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.714411 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3000b819a4cc4960909b98febd400631. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.714740 890 tablet_service.cc:1467] Processing CreateTablet for tablet c00aaf81b34445abb7502700f01d4f3f (DEFAULT_TABLE table=default.Table_ea6be4d48d6041f493f1bf85432bfb0e [id=0430588a51ed49ae86290eaab54f79dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.715160 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00aaf81b34445abb7502700f01d4f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.716336 898 tablet_service.cc:1467] Processing CreateTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 (DEFAULT_TABLE table=default.Table_d59d31f7bc514ff38a99d27cbe8170f9 [id=5514c15f7b8c406288bec6394be1d99b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.716641 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c6f1833ff1453f90a6ec6435248a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.717010 899 tablet_service.cc:1467] Processing CreateTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c (DEFAULT_TABLE table=default.Table_4792b8689a12402f9946e87ad47dcbbb [id=ec2d9d93307747a7b752255f2008500e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.717401 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acca65ecee94f8cbe3f2b89e744aa6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.714421 895 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab7ca098694464faec002cec05558ac (DEFAULT_TABLE table=default.Table_98434239d23e44ad848ad3a212ae9aac [id=7e91cd7d62904c858af46544ca9761a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.721133 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab7ca098694464faec002cec05558ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.721094 892 tablet_service.cc:1467] Processing CreateTablet for tablet c03524194a7b4e34a20424091e308ac4 (DEFAULT_TABLE table=default.Table_e182e0b0c2b6434eb2acce63b8935309 [id=d89b416c6f8d4417a6ccb26b443b2ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.721509 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03524194a7b4e34a20424091e308ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.734001 1160 ts_tablet_manager.cc:1918] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.734150 1160 log.cc:1198] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/26f5b4af510942c8b50838df3bc8d8fc
I20250109 20:52:44.734541 1160 ts_tablet_manager.cc:1939] T 26f5b4af510942c8b50838df3bc8d8fc P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:44.735001 1033 tablet_service.cc:1467] Processing CreateTablet for tablet c03524194a7b4e34a20424091e308ac4 (DEFAULT_TABLE table=default.Table_e182e0b0c2b6434eb2acce63b8935309 [id=d89b416c6f8d4417a6ccb26b443b2ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.735453 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03524194a7b4e34a20424091e308ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.735496 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c00aaf81b34445abb7502700f01d4f3f (DEFAULT_TABLE table=default.Table_ea6be4d48d6041f493f1bf85432bfb0e [id=0430588a51ed49ae86290eaab54f79dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.735550 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c (DEFAULT_TABLE table=default.Table_4792b8689a12402f9946e87ad47dcbbb [id=ec2d9d93307747a7b752255f2008500e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.735778 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00aaf81b34445abb7502700f01d4f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.736006 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acca65ecee94f8cbe3f2b89e744aa6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.741086 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 3000b819a4cc4960909b98febd400631 (DEFAULT_TABLE table=default.Table_77d18f6149124e54b0a321ccd3bd8fab [id=2d996a757dae433a9853245f2ceb1f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.741482 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3000b819a4cc4960909b98febd400631. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.744596 1160 tablet_replica.cc:331] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:44.744815 1160 raft_consensus.cc:2238] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.745126 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 (DEFAULT_TABLE table=default.Table_d59d31f7bc514ff38a99d27cbe8170f9 [id=5514c15f7b8c406288bec6394be1d99b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.745496 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c6f1833ff1453f90a6ec6435248a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.745738 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 26f5b4af510942c8b50838df3bc8d8fc (table default.Table_3eef1bf244124c8d8d9bda7d0699a6f7 [id=0048f4157ceb4c098d924be9004eeaa1]) successfully deleted
I20250109 20:52:44.746250 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 660e60f368cb4e25b12ab3d7b2e314b1 (table default.Table_b7ff81513ee94c0bbc84bb3eeea7f144 [id=028851fe80724d3cb4402101784bd12a]). Expected version 1 got 0
I20250109 20:52:44.749037 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab7ca098694464faec002cec05558ac (DEFAULT_TABLE table=default.Table_98434239d23e44ad848ad3a212ae9aac [id=7e91cd7d62904c858af46544ca9761a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.749492 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab7ca098694464faec002cec05558ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.749629 1160 raft_consensus.cc:2267] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.750288 1160 ts_tablet_manager.cc:1905] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.750473 1240 raft_consensus.cc:491] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.750613 1240 raft_consensus.cc:513] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.751307 1240 leader_election.cc:290] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:44.752115 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d7cead4f4149ae8de48eee2cfd8679" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:44.752151 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d7cead4f4149ae8de48eee2cfd8679" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:44.752604 990 leader_election.cc:343] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.753037 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 5e90cd406e394166b3e8dedebf7fd0fc (DEFAULT_TABLE table=default.Table_a97de1805a3143398c7a97e52e5d2e64 [id=e39eeed4512c4637b813b7ee38d4318d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.753454 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e90cd406e394166b3e8dedebf7fd0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.758018 748 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab7ca098694464faec002cec05558ac (DEFAULT_TABLE table=default.Table_98434239d23e44ad848ad3a212ae9aac [id=7e91cd7d62904c858af46544ca9761a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.758458 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab7ca098694464faec002cec05558ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.761011 744 tablet_service.cc:1467] Processing CreateTablet for tablet c00aaf81b34445abb7502700f01d4f3f (DEFAULT_TABLE table=default.Table_ea6be4d48d6041f493f1bf85432bfb0e [id=0430588a51ed49ae86290eaab54f79dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.761412 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00aaf81b34445abb7502700f01d4f3f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:44.763593 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250109 20:52:44.765007 745 tablet_service.cc:1467] Processing CreateTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 (DEFAULT_TABLE table=default.Table_d59d31f7bc514ff38a99d27cbe8170f9 [id=5514c15f7b8c406288bec6394be1d99b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.765372 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c6f1833ff1453f90a6ec6435248a02. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:44.753014 989 leader_election.cc:343] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.769099 989 leader_election.cc:304] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:44.769622 1240 raft_consensus.cc:2743] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:44.772293 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 660e60f368cb4e25b12ab3d7b2e314b1 consensus state has no leader
I20250109 20:52:44.772709 1165 tablet_bootstrap.cc:492] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.772747 1176 tablet_bootstrap.cc:492] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.772861 1176 ts_tablet_manager.cc:1397] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.147s user 0.021s sys 0.022s
I20250109 20:52:44.772877 1165 ts_tablet_manager.cc:1397] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.136s user 0.050s sys 0.069s
W20250109 20:52:44.773658 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c00aaf81b34445abb7502700f01d4f3f consensus state has no leader
I20250109 20:52:44.773564 1165 raft_consensus.cc:357] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.773583 1176 raft_consensus.cc:357] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.773774 1176 raft_consensus.cc:383] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.773783 1165 raft_consensus.cc:383] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.773850 1165 raft_consensus.cc:738] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.773869 1176 raft_consensus.cc:738] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.774080 1176 consensus_queue.cc:260] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.774454 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cfa3005fdd40709a630d5bb80564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.774606 1176 ts_tablet_manager.cc:1428] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:44.774678 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.774838 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ab00e387fbc843b2841bef2d3289df34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.774878 1176 tablet_bootstrap.cc:492] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.775074 1166 tablet_replica.cc:331] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.775238 1166 raft_consensus.cc:2238] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.775403 1166 raft_consensus.cc:2267] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.776278 1166 ts_tablet_manager.cc:1905] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.777236 1165 consensus_queue.cc:260] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.778784 1160 ts_tablet_manager.cc:1918] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.778910 1160 log.cc:1198] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/30ae6c4e835048bd8deed5ac9c4ae07a
I20250109 20:52:44.779230 1160 ts_tablet_manager.cc:1939] T 30ae6c4e835048bd8deed5ac9c4ae07a P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:44.779536 1165 ts_tablet_manager.cc:1428] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250109 20:52:44.779798 1165 tablet_bootstrap.cc:492] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:44.780131 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 30ae6c4e835048bd8deed5ac9c4ae07a (table default.Table_a6d4d40a237548c384fd897eb207903c [id=6fe0e6310b9f44a8b3b9558fec22c2a6]) successfully deleted
I20250109 20:52:44.781308 1165 tablet_bootstrap.cc:654] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.781409 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fb6aeedc77452f99c46f03d6d8d089 with delete_type TABLET_DATA_DELETED (replaced by tablet 779153d4174b4887b5dc0bb355cb4a9d) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.782974 1176 tablet_bootstrap.cc:654] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.786106 1160 tablet_replica.cc:331] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:44.786329 1160 raft_consensus.cc:2238] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.786528 1160 raft_consensus.cc:2267] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.788842 1160 ts_tablet_manager.cc:1905] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.798804 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f580256f4281489b85b9f46fcd381c5f" } new_table_name: "default.Table_5d1e5a64991f49a48a7b4286fc1ffe5f" modify_external_catalogs: true
I20250109 20:52:44.799538 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet ab593a602d61495eb7b323903ae610b6 (table default.Table_ab5a0c56116047958eb48524522f7b27 [id=df6e0f6dba1942d9b232a020d5d2b4d9]). Expected version 3 got 0
I20250109 20:52:44.800765 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0f1b6adda4ab4df18b08fb9f430907e8" } modify_external_catalogs: true
I20250109 20:52:44.800951 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0f1b6adda4ab4df18b08fb9f430907e8" } modify_external_catalogs: true
I20250109 20:52:44.793793 1166 ts_tablet_manager.cc:1918] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.805002 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_58060dd58dce4f7a8afa4d8c891c3ee3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.805486 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58060dd58dce4f7a8afa4d8c891c3ee3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.806782 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c9fc2d115b114c17af39f021ef808525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:44.808398 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_759654c8879645ee84625859a22e72b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:44.809363 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 03b47ee787ca4dc5a4a56ae582d15b8b
I20250109 20:52:44.811808 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1925a82eb711409ab2c5429c61edf9d8
I20250109 20:52:44.813268 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 32eb4dfcf96c4a05a55082e90ae04467
W20250109 20:52:44.814437 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9fc2d115b114c17af39f021ef808525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.816361 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2acf186aebe7415881f4304654dc6a83" } new_table_name: "default.Table_ea3137b2ba3d44e3a715f18eb25dca01" modify_external_catalogs: true
I20250109 20:52:44.822003 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:44.823702 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11d303cd42a2428fa17f29b48c4f64f5 consensus state has no leader
I20250109 20:52:44.824360 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ab00e387fbc843b2841bef2d3289df34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.825093 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.825065 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7771b3421a9449929095a6e25dcbf470"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:44.825237 1158 tablet_bootstrap.cc:492] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:44.825354 1158 ts_tablet_manager.cc:1397] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.167s user 0.007s sys 0.027s
W20250109 20:52:44.825539 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7771b3421a9449929095a6e25dcbf470 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.826136 1158 raft_consensus.cc:357] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.826562 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ab5a0c56116047958eb48524522f7b27" } modify_external_catalogs: true
I20250109 20:52:44.826715 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ab5a0c56116047958eb48524522f7b27" } modify_external_catalogs: true
I20250109 20:52:44.826301 1158 raft_consensus.cc:383] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.827605 1158 raft_consensus.cc:738] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.827836 1158 consensus_queue.cc:260] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.828797 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_609b1c68b73f4e99ad6afa14ee14ad50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.829227 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_609b1c68b73f4e99ad6afa14ee14ad50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.830312 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_651b2a8068904467a07c19ff16909b5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.830716 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_651b2a8068904467a07c19ff16909b5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.834609 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.835834 1158 ts_tablet_manager.cc:1428] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:44.836050 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b9feb9de9dc4d22bafc0f7707943113 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.836145 1158 tablet_bootstrap.cc:492] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:44.809377 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_759654c8879645ee84625859a22e72b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.838126 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.812669 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.814222 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.818946 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1925a82eb711409ab2c5429c61edf9d8) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.819550 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.823139 1240 raft_consensus.cc:491] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.839972 1240 raft_consensus.cc:513] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.840590 1240 leader_election.cc:290] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:44.823361 1187 raft_consensus.cc:491] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.840782 1187 raft_consensus.cc:513] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.841250 1187 leader_election.cc:290] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:44.842125 1166 log.cc:1198] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/34cfa3005fdd40709a630d5bb80564db
I20250109 20:52:44.842500 1166 ts_tablet_manager.cc:1939] T 34cfa3005fdd40709a630d5bb80564db P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.843407 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 34cfa3005fdd40709a630d5bb80564db (table default.Table_5c35bca9b68e42bb9075f51687f74d15 [id=261eb60785fc4d11909ad99fe7cb53d0]) successfully deleted
I20250109 20:52:44.843680 1166 tablet_replica.cc:331] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.843777 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab00e387fbc843b2841bef2d3289df34" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:44.843878 1166 raft_consensus.cc:2238] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:44.843986 778 raft_consensus.cc:2463] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:44.844255 1166 pending_rounds.cc:62] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Trying to abort 1 pending ops.
I20250109 20:52:44.844440 1166 pending_rounds.cc:69] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523630837919744 op_type: NO_OP noop_request { }
I20250109 20:52:44.844583 1166 raft_consensus.cc:2883] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:44.844672 1166 raft_consensus.cc:2267] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.845515 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.845814 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.846103 1191 raft_consensus.cc:491] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.846452 1191 raft_consensus.cc:513] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.847052 1191 leader_election.cc:290] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:44.847864 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab00e387fbc843b2841bef2d3289df34" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:44.848434 743 tablet_service.cc:1467] Processing CreateTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c (DEFAULT_TABLE table=default.Table_4792b8689a12402f9946e87ad47dcbbb [id=ec2d9d93307747a7b752255f2008500e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.849277 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded8e3a447c84b9b8ec4ccd70199ccd5" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:44.846287 990 leader_election.cc:304] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:44.851238 1187 raft_consensus.cc:2798] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:44.851328 1187 raft_consensus.cc:491] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:44.851404 1187 raft_consensus.cc:3054] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:44.853880 1187 raft_consensus.cc:513] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.854722 1187 leader_election.cc:290] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:52:44.854857 989 leader_election.cc:343] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:44.855340 989 leader_election.cc:343] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.855376 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_56c8a1eb8409437c856c4af9ece36c61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.855891 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56c8a1eb8409437c856c4af9ece36c61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.857661 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e951676901de4adab1fb93a2d265da4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:44.858032 1166 ts_tablet_manager.cc:1905] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:44.858305 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e951676901de4adab1fb93a2d265da4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.858947 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 779153d4174b4887b5dc0bb355cb4a9d
I20250109 20:52:44.859760 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.860137 455 master_service.cc:946] ReplaceTablet: received request to replace tablet b0fda753bbc940f7834507345f67651c from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.860370 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5d6390d21a0241e1bd37f6158126ff0e" } new_table_name: "default.Table_aaff98763d0f4eff96db3f5d2d85ba2a" modify_external_catalogs: true
I20250109 20:52:44.860630 455 master_service.cc:946] ReplaceTablet: received request to replace tablet cc174d6bd9f44eaf9f66e54693597601 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.863720 1166 ts_tablet_manager.cc:1918] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:44.863867 1166 log.cc:1198] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/eeedf2e359c94a96ba6a518cefbb1fbe
I20250109 20:52:44.864310 1166 ts_tablet_manager.cc:1939] T eeedf2e359c94a96ba6a518cefbb1fbe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.865940 1166 tablet_replica.cc:331] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.866369 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet eeedf2e359c94a96ba6a518cefbb1fbe (table default.Table_e1e2df6604b3481583f4036b3aee752e [id=ae10da31c7cf49789ddf4c05c169c105]) successfully deleted
I20250109 20:52:44.866298 1166 raft_consensus.cc:2238] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.866768 1166 raft_consensus.cc:2267] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.867660 1166 ts_tablet_manager.cc:1905] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.867645 742 tablet_service.cc:1514] Processing DeleteTablet for tablet bd445a1adba34529aa3f7a44b991b177 with delete_type TABLET_DATA_DELETED (replaced by tablet 5496b2f965064f7788ebd74c62d1240a) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.867856 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.868860 455 master_service.cc:946] ReplaceTablet: received request to replace tablet eeedf2e359c94a96ba6a518cefbb1fbe from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.870007 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded8e3a447c84b9b8ec4ccd70199ccd5" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:44.870797 1176 tablet_bootstrap.cc:492] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.870950 1176 ts_tablet_manager.cc:1397] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.096s user 0.000s sys 0.010s
I20250109 20:52:44.871176 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ab593a602d61495eb7b323903ae610b6
W20250109 20:52:44.871199 990 leader_election.cc:343] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.871315 990 leader_election.cc:304] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:44.871963 1187 raft_consensus.cc:2743] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:44.871904 1176 raft_consensus.cc:357] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.872175 1176 raft_consensus.cc:383] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.872257 1176 raft_consensus.cc:738] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.872331 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet ab593a602d61495eb7b323903ae610b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.872578 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.872514 1176 consensus_queue.cc:260] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:44.873875 1176 ts_tablet_manager.cc:1428] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:44.874193 1176 tablet_bootstrap.cc:492] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.874269 1157 ts_tablet_manager.cc:1918] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.874377 1157 log.cc:1198] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f6a1a6a0a1f4411aa0b57481b6291808
I20250109 20:52:44.877285 1176 tablet_bootstrap.cc:654] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.877934 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ab593a602d61495eb7b323903ae610b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.878082 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3000b819a4cc4960909b98febd400631 (DEFAULT_TABLE table=default.Table_77d18f6149124e54b0a321ccd3bd8fab [id=2d996a757dae433a9853245f2ceb1f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.878154 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2b9feb9de9dc4d22bafc0f7707943113
I20250109 20:52:44.878283 756 tablet_service.cc:1467] Processing CreateTablet for tablet c03524194a7b4e34a20424091e308ac4 (DEFAULT_TABLE table=default.Table_e182e0b0c2b6434eb2acce63b8935309 [id=d89b416c6f8d4417a6ccb26b443b2ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.878578 450 catalog_manager.cc:6596] ReplaceTablet: tablet 2b9feb9de9dc4d22bafc0f7707943113 deleted and replaced by tablet e64207928ef141118e7fc8fe7bb2379b
I20250109 20:52:44.878535 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab00e387fbc843b2841bef2d3289df34" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:44.878716 776 raft_consensus.cc:3054] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:44.881636 776 raft_consensus.cc:2463] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:44.882174 990 leader_election.cc:304] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
W20250109 20:52:44.882362 990 raft_consensus.cc:2681] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:52:44.884312 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 with delete_type TABLET_DATA_DELETED (replaced by tablet e64207928ef141118e7fc8fe7bb2379b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.885807 1166 ts_tablet_manager.cc:1918] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.886248 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 with delete_type TABLET_DATA_DELETED (replaced by tablet e64207928ef141118e7fc8fe7bb2379b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.887598 1166 log.cc:1198] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ab00e387fbc843b2841bef2d3289df34
I20250109 20:52:44.888216 1166 ts_tablet_manager.cc:1939] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.889595 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ab00e387fbc843b2841bef2d3289df34 (table default.Table_49a9931b8903487db0856c10d60fc1e9 [id=8e13eabc06da4067aa8108f30c8d323d]) successfully deleted
I20250109 20:52:44.889963 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.890552 1158 tablet_bootstrap.cc:654] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:44.891026 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1925a82eb711409ab2c5429c61edf9d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1925a82eb711409ab2c5429c61edf9d8 already in progress: creating tablet
W20250109 20:52:44.891857 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 32eb4dfcf96c4a05a55082e90ae04467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32eb4dfcf96c4a05a55082e90ae04467 already in progress: creating tablet
I20250109 20:52:44.892203 1176 tablet_bootstrap.cc:492] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.892333 1166 tablet_replica.cc:331] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.892419 1176 ts_tablet_manager.cc:1397] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.008s
I20250109 20:52:44.892518 1166 raft_consensus.cc:2238] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.892722 1166 raft_consensus.cc:2267] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.893724 1176 raft_consensus.cc:357] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.893985 1176 raft_consensus.cc:383] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.894162 1176 raft_consensus.cc:738] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.894148 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab00e387fbc843b2841bef2d3289df34" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:44.893963 1166 ts_tablet_manager.cc:1905] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.894590 1176 consensus_queue.cc:260] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.895083 1176 ts_tablet_manager.cc:1428] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:44.895373 1176 tablet_bootstrap.cc:492] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:44.897655 854 leader_election.cc:343] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: d2aacfc7eea3490dbeb9d5c82d48766d
I20250109 20:52:44.893316 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.893466 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acca65ecee94f8cbe3f2b89e744aa6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.901300 1176 tablet_bootstrap.cc:654] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.904879 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03524194a7b4e34a20424091e308ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.904949 1166 ts_tablet_manager.cc:1918] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.907101 1166 log.cc:1198] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ded8e3a447c84b9b8ec4ccd70199ccd5
I20250109 20:52:44.907680 1166 ts_tablet_manager.cc:1939] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.907713 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3000b819a4cc4960909b98febd400631. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:44.908408 989 leader_election.cc:343] T ab00e387fbc843b2841bef2d3289df34 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:44.908967 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 50057us
I20250109 20:52:44.909993 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ded8e3a447c84b9b8ec4ccd70199ccd5 (table default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]) successfully deleted
I20250109 20:52:44.910818 1166 tablet_replica.cc:331] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.911240 1166 raft_consensus.cc:2238] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.911625 1166 raft_consensus.cc:2267] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.911466 1176 tablet_bootstrap.cc:492] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.911943 452 master_service.cc:946] ReplaceTablet: received request to replace tablet ded8e3a447c84b9b8ec4ccd70199ccd5 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:44.912042 1176 ts_tablet_manager.cc:1397] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.001s
I20250109 20:52:44.912564 1166 ts_tablet_manager.cc:1905] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:44.912911 854 leader_election.cc:343] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: d2aacfc7eea3490dbeb9d5c82d48766d
I20250109 20:52:44.913045 854 leader_election.cc:304] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:44.896939 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cfa3005fdd40709a630d5bb80564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:44.912828 1176 raft_consensus.cc:357] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.913533 1176 raft_consensus.cc:383] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.913666 1176 raft_consensus.cc:738] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.900213 1157 ts_tablet_manager.cc:1939] T f6a1a6a0a1f4411aa0b57481b6291808 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:44.913997 1176 consensus_queue.cc:260] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.914125 1191 raft_consensus.cc:2743] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:44.914599 1176 ts_tablet_manager.cc:1428] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:44.914942 1176 tablet_bootstrap.cc:492] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.915081 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f6a1a6a0a1f4411aa0b57481b6291808 (table default.Table_dab2393d93b14ec79d96807a37360e33 [id=fcec4978e3584a9cabd5641b5ffb7340]) successfully deleted
I20250109 20:52:44.917275 1176 tablet_bootstrap.cc:654] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.917865 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 with delete_type TABLET_DATA_DELETED (replaced by tablet e64207928ef141118e7fc8fe7bb2379b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.918331 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_78b865cbf0f2431c9ee05f94caddda22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:44.918953 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78b865cbf0f2431c9ee05f94caddda22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:44.919952 1160 ts_tablet_manager.cc:1918] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.920106 1160 log.cc:1198] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/e5d6dc79bf1e48ee9cf2bbb70463741d
I20250109 20:52:44.920943 1160 ts_tablet_manager.cc:1939] T e5d6dc79bf1e48ee9cf2bbb70463741d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:44.921185 1157 tablet_replica.cc:331] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:44.922443 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet e5d6dc79bf1e48ee9cf2bbb70463741d (table default.Table_f9f28decdfb64455809f7b283552a4a5 [id=ae7fb9326c2c49a593d1111d45ea424f]) successfully deleted
I20250109 20:52:44.922737 1157 raft_consensus.cc:2238] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.922924 1157 raft_consensus.cc:2267] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.923053 1166 ts_tablet_manager.cc:1918] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.923449 1157 ts_tablet_manager.cc:1905] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.924196 1160 tablet_replica.cc:331] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:44.924414 1160 raft_consensus.cc:2238] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.924587 1160 raft_consensus.cc:2267] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.925069 1160 ts_tablet_manager.cc:1905] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.925477 1166 log.cc:1198] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/68aa4a05c36e44b0aa6ee03780d2a815
I20250109 20:52:44.925974 1166 ts_tablet_manager.cc:1939] T 68aa4a05c36e44b0aa6ee03780d2a815 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.927277 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 68aa4a05c36e44b0aa6ee03780d2a815 (table default.Table_27fb907c2b28435db3522088b3d352fa [id=4e6d2e7acdc84a8fa2d3bd56e8437974]) successfully deleted
W20250109 20:52:44.928624 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 03b47ee787ca4dc5a4a56ae582d15b8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03b47ee787ca4dc5a4a56ae582d15b8b already in progress: creating tablet
W20250109 20:52:44.929574 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 779153d4174b4887b5dc0bb355cb4a9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 779153d4174b4887b5dc0bb355cb4a9d already in progress: creating tablet
I20250109 20:52:44.930114 1166 tablet_replica.cc:331] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.930469 1166 raft_consensus.cc:2238] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.930789 1176 tablet_bootstrap.cc:492] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.930990 1176 ts_tablet_manager.cc:1397] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20250109 20:52:44.931084 1166 raft_consensus.cc:2267] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:44.931633 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff4f020c5ba643aeb11f8e08d748be46 consensus state has no leader
I20250109 20:52:44.931972 1166 ts_tablet_manager.cc:1905] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.931952 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ab593a602d61495eb7b323903ae610b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.932276 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.933861 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170]). Expected version 2 got 0
I20250109 20:52:44.931972 1176 raft_consensus.cc:357] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.936544 1176 raft_consensus.cc:383] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.936641 1176 raft_consensus.cc:738] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.936882 1176 consensus_queue.cc:260] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.937263 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 with delete_type TABLET_DATA_DELETED (replaced by tablet cba0ffb1b394431ba30918911cdee7b7) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.937889 1176 ts_tablet_manager.cc:1428] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:52:44.938292 1176 tablet_bootstrap.cc:492] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.938498 1165 tablet_bootstrap.cc:492] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.938683 1165 ts_tablet_manager.cc:1397] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.159s user 0.028s sys 0.053s
I20250109 20:52:44.939671 1165 raft_consensus.cc:357] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.939975 1165 raft_consensus.cc:383] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.940129 1165 raft_consensus.cc:738] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.940436 1165 consensus_queue.cc:260] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.941016 1165 ts_tablet_manager.cc:1428] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:44.941361 1165 tablet_bootstrap.cc:492] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:44.942982 1165 tablet_bootstrap.cc:654] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.943272 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 175bf6c335a8405c97b296cf4718610b with delete_type TABLET_DATA_DELETED (replaced by tablet 11d303cd42a2428fa17f29b48c4f64f5) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:44.944476 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 540531463c184b9abf226d7487939e60 (DEFAULT_TABLE table=default.Table_7609a405eec042e4982438d13650f8a4 [id=59a36af89f8f4df0ace607f3190236ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.944816 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 (DEFAULT_TABLE table=default.Table_834dbde1ccad4d8bb60efe0ae6ecf469 [id=70a6ff8c791e41038b082b9004740857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.944963 1176 tablet_bootstrap.cc:654] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.944983 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 540531463c184b9abf226d7487939e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.945289 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a89a0b47d10414d9b6a9123af1f4ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.946440 896 tablet_service.cc:1514] Processing DeleteTablet for tablet ab00e387fbc843b2841bef2d3289df34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:44.942919 891 tablet_service.cc:1467] Processing CreateTablet for tablet 540531463c184b9abf226d7487939e60 (DEFAULT_TABLE table=default.Table_7609a405eec042e4982438d13650f8a4 [id=59a36af89f8f4df0ace607f3190236ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.951134 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 540531463c184b9abf226d7487939e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.952261 896 tablet_service.cc:1467] Processing CreateTablet for tablet 2f494150713b4dde9399d53d57567983 (DEFAULT_TABLE table=default.Table_7d5333c1118343aeb596d21852bbedbf [id=22e6d25d81154e46961b3a6322faf2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.952772 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f494150713b4dde9399d53d57567983. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.953050 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 (DEFAULT_TABLE table=default.Table_834dbde1ccad4d8bb60efe0ae6ecf469 [id=70a6ff8c791e41038b082b9004740857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.953502 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a89a0b47d10414d9b6a9123af1f4ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.963653 1166 ts_tablet_manager.cc:1918] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.964664 757 tablet_service.cc:1467] Processing CreateTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 (DEFAULT_TABLE table=default.Table_834dbde1ccad4d8bb60efe0ae6ecf469 [id=70a6ff8c791e41038b082b9004740857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.964624 1176 tablet_bootstrap.cc:492] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:44.965420 1166 log.cc:1198] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ab593a602d61495eb7b323903ae610b6
I20250109 20:52:44.965595 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a89a0b47d10414d9b6a9123af1f4ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.965493 1176 ts_tablet_manager.cc:1397] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20250109 20:52:44.966159 1166 ts_tablet_manager.cc:1939] T ab593a602d61495eb7b323903ae610b6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:44.966553 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 2f494150713b4dde9399d53d57567983 (DEFAULT_TABLE table=default.Table_7d5333c1118343aeb596d21852bbedbf [id=22e6d25d81154e46961b3a6322faf2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.966470 1176 raft_consensus.cc:357] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.969141 1176 raft_consensus.cc:383] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.969228 1176 raft_consensus.cc:738] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.969478 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f494150713b4dde9399d53d57567983. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.969484 1176 consensus_queue.cc:260] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.969102 1157 ts_tablet_manager.cc:1918] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.970060 1176 ts_tablet_manager.cc:1428] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:44.970350 1176 tablet_bootstrap.cc:492] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:44.973569 1176 tablet_bootstrap.cc:654] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:44.972684 896 tablet_service.cc:1467] Processing CreateTablet for tablet cba68160be6145ddbbd2019c9b5f0076 (DEFAULT_TABLE table=default.Table_e03d934cb25e4e96b3c9670d7698253a [id=f556fe2500bb47efba4056c36ca34823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.977265 748 tablet_service.cc:1467] Processing CreateTablet for tablet 2f494150713b4dde9399d53d57567983 (DEFAULT_TABLE table=default.Table_7d5333c1118343aeb596d21852bbedbf [id=22e6d25d81154e46961b3a6322faf2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.977461 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ab593a602d61495eb7b323903ae610b6 (table default.Table_ab5a0c56116047958eb48524522f7b27 [id=df6e0f6dba1942d9b232a020d5d2b4d9]) successfully deleted
I20250109 20:52:44.977768 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba68160be6145ddbbd2019c9b5f0076. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:44.978827 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2b9feb9de9dc4d22bafc0f7707943113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9feb9de9dc4d22bafc0f7707943113 already in progress: creating tablet
I20250109 20:52:44.979384 1166 tablet_replica.cc:331] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:44.981197 1157 log.cc:1198] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/89fb6aeedc77452f99c46f03d6d8d089
I20250109 20:52:44.982841 1160 ts_tablet_manager.cc:1918] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:44.983992 1160 log.cc:1198] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/6bea28fb446e49669133e21b02c416ad
I20250109 20:52:44.981197 746 tablet_service.cc:1467] Processing CreateTablet for tablet cba68160be6145ddbbd2019c9b5f0076 (DEFAULT_TABLE table=default.Table_e03d934cb25e4e96b3c9670d7698253a [id=f556fe2500bb47efba4056c36ca34823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.984181 899 tablet_service.cc:1467] Processing CreateTablet for tablet 70b48900f57f47128e31b27275b1489c (DEFAULT_TABLE table=default.Table_e4f018070aac4f4298a611b89b0e5f6a [id=b9ba88e68399420c8bffa0407b369aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.984627 1166 raft_consensus.cc:2238] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:44.984845 1166 raft_consensus.cc:2267] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:44.984894 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b48900f57f47128e31b27275b1489c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.984946 1160 ts_tablet_manager.cc:1939] T 6bea28fb446e49669133e21b02c416ad P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:44.983786 1017 tablet_service.cc:1467] Processing CreateTablet for tablet cba68160be6145ddbbd2019c9b5f0076 (DEFAULT_TABLE table=default.Table_e03d934cb25e4e96b3c9670d7698253a [id=f556fe2500bb47efba4056c36ca34823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.985232 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba68160be6145ddbbd2019c9b5f0076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.985828 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 6bea28fb446e49669133e21b02c416ad (table default.Table_5ab3d2823bee4f739f414db53a1a6cd9 [id=06cecc207bb746dd9b0cb0f6eac4017e]) successfully deleted
I20250109 20:52:44.987238 893 tablet_service.cc:1467] Processing CreateTablet for tablet dae574bf93d649099252e9e7b09b748c (DEFAULT_TABLE table=default.Table_655ff82c5466442c818f93c7d408c157 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.987531 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dae574bf93d649099252e9e7b09b748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.981385 745 tablet_service.cc:1467] Processing CreateTablet for tablet 540531463c184b9abf226d7487939e60 (DEFAULT_TABLE table=default.Table_7609a405eec042e4982438d13650f8a4 [id=59a36af89f8f4df0ace607f3190236ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:44.983317 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f494150713b4dde9399d53d57567983. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:44.989574 1165 tablet_bootstrap.cc:492] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:44.989763 1165 ts_tablet_manager.cc:1397] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.002s
I20250109 20:52:44.990449 1165 raft_consensus.cc:357] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.990630 1165 raft_consensus.cc:383] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:44.990725 1165 raft_consensus.cc:738] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:44.991366 1165 consensus_queue.cc:260] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:44.991931 1165 ts_tablet_manager.cc:1428] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:44.992228 1166 ts_tablet_manager.cc:1905] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:44.993428 1165 tablet_bootstrap.cc:492] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:44.993502 1187 raft_consensus.cc:491] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:44.993639 1187 raft_consensus.cc:513] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:44.994302 1187 leader_election.cc:290] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:44.994982 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e06e64448a4fb2bf41fc6e3d0361a6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:44.995623 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 660e60f368cb4e25b12ab3d7b2e314b1 (table default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]). Expected version 2 got 0
I20250109 20:52:44.996073 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c9fc2d115b114c17af39f021ef808525" } new_table_name: "default.Table_e740022397a8451e830a159fa021e461" modify_external_catalogs: true
I20250109 20:52:44.998488 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e182e0b0c2b6434eb2acce63b8935309" } new_table_name: "default.Table_6db1fc2ef56942dc9e962b951f82a995" modify_external_catalogs: true
W20250109 20:52:44.998603 989 leader_election.cc:343] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:44.998919 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 660e60f368cb4e25b12ab3d7b2e314b1 (table default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]). Expected version 2 got 0
I20250109 20:52:45.001957 744 tablet_service.cc:1467] Processing CreateTablet for tablet 70b48900f57f47128e31b27275b1489c (DEFAULT_TABLE table=default.Table_e4f018070aac4f4298a611b89b0e5f6a [id=b9ba88e68399420c8bffa0407b369aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.003400 1026 tablet_service.cc:1467] Processing CreateTablet for tablet dae574bf93d649099252e9e7b09b748c (DEFAULT_TABLE table=default.Table_655ff82c5466442c818f93c7d408c157 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.003768 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dae574bf93d649099252e9e7b09b748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.004104 1268 raft_consensus.cc:491] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.004254 1268 raft_consensus.cc:513] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.004894 1268 leader_election.cc:290] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:45.005812 713 leader_election.cc:343] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: bd445a1adba34529aa3f7a44b991b177
I20250109 20:52:45.006230 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd445a1adba34529aa3f7a44b991b177" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:45.006703 712 leader_election.cc:343] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.006855 712 leader_election.cc:304] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:45.007596 1268 raft_consensus.cc:2743] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:45.008237 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 402280f75e51463a865c0fb521d0fd8a has no consensus state
I20250109 20:52:45.009961 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 70b48900f57f47128e31b27275b1489c (DEFAULT_TABLE table=default.Table_e4f018070aac4f4298a611b89b0e5f6a [id=b9ba88e68399420c8bffa0407b369aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.010772 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b48900f57f47128e31b27275b1489c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:45.011619 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c03524194a7b4e34a20424091e308ac4 consensus state has no leader
I20250109 20:52:45.013243 1160 tablet_replica.cc:331] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.013432 1165 tablet_bootstrap.cc:654] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.014634 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e06e64448a4fb2bf41fc6e3d0361a6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:45.014866 891 tablet_service.cc:1467] Processing CreateTablet for tablet b77c8d6adbf44375aff2782a83643255 (DEFAULT_TABLE table=default.Table_a5bb2f6d26e3423e93672a1954c238a7 [id=43c66cf1f4554c79b35bd2689050a945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.014993 1160 raft_consensus.cc:2238] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.015282 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba68160be6145ddbbd2019c9b5f0076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.015484 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b77c8d6adbf44375aff2782a83643255. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.015672 1160 raft_consensus.cc:2267] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.015967 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b48900f57f47128e31b27275b1489c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.018852 1160 ts_tablet_manager.cc:1905] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.021046 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9d14aaae5e2d40a3842ad98fabf33015" } modify_external_catalogs: true
I20250109 20:52:45.021235 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9d14aaae5e2d40a3842ad98fabf33015" } modify_external_catalogs: true
I20250109 20:52:45.022553 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d78dfd28e10e45549a210fad737f3627"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.023000 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d78dfd28e10e45549a210fad737f3627 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.023298 742 tablet_service.cc:1467] Processing CreateTablet for tablet dae574bf93d649099252e9e7b09b748c (DEFAULT_TABLE table=default.Table_655ff82c5466442c818f93c7d408c157 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.021119 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d8c45162ae564eb0b502f22e5bfe50bd" } modify_external_catalogs: true
I20250109 20:52:45.024346 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d8c45162ae564eb0b502f22e5bfe50bd" } modify_external_catalogs: true
I20250109 20:52:45.025535 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1bca442744ee4e04a46973fb21434cae" } modify_external_catalogs: true
I20250109 20:52:45.025683 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1bca442744ee4e04a46973fb21434cae" } modify_external_catalogs: true
W20250109 20:52:45.025913 990 leader_election.cc:343] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.026039 990 leader_election.cc:304] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:45.026638 1187 raft_consensus.cc:2743] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.027951 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 (DEFAULT_TABLE table=default.Table_c2a802cfb927494f92e603090370e543 [id=dccf417b39fc4d2d87812e340ecd842e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.028761 1165 tablet_bootstrap.cc:492] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:45.029080 1165 ts_tablet_manager.cc:1397] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.001s
I20250109 20:52:45.029345 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb563bd6802473fafedd08bb10d15c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.030653 1165 raft_consensus.cc:357] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.031026 1165 raft_consensus.cc:383] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.031200 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 540531463c184b9abf226d7487939e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.031167 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8730181edd7f48769b95dd508118d1df" } new_table_name: "default.Table_332207f19edb4562826601b0dbe41dce" modify_external_catalogs: true
I20250109 20:52:45.031188 1165 raft_consensus.cc:738] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.031606 1165 consensus_queue.cc:260] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.032135 758 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 (DEFAULT_TABLE table=default.Table_c2a802cfb927494f92e603090370e543 [id=dccf417b39fc4d2d87812e340ecd842e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.035969 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b77c8d6adbf44375aff2782a83643255 (DEFAULT_TABLE table=default.Table_a5bb2f6d26e3423e93672a1954c238a7 [id=43c66cf1f4554c79b35bd2689050a945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.036633 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b77c8d6adbf44375aff2782a83643255. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.039678 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e6cc2c7b12f945ea8307a187e1553226 (DEFAULT_TABLE table=default.Table_18addc41a8074f018e26f238d2e7ca70 [id=e0737f359bfa4154b4fb71617efc829f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.040467 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6cc2c7b12f945ea8307a187e1553226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.033699 1165 ts_tablet_manager.cc:1428] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250109 20:52:45.037375 899 tablet_service.cc:1467] Processing CreateTablet for tablet e6cc2c7b12f945ea8307a187e1553226 (DEFAULT_TABLE table=default.Table_18addc41a8074f018e26f238d2e7ca70 [id=e0737f359bfa4154b4fb71617efc829f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.042197 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6cc2c7b12f945ea8307a187e1553226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.043063 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dae574bf93d649099252e9e7b09b748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.044302 1165 tablet_bootstrap.cc:492] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.046855 757 tablet_service.cc:1467] Processing CreateTablet for tablet b77c8d6adbf44375aff2782a83643255 (DEFAULT_TABLE table=default.Table_a5bb2f6d26e3423e93672a1954c238a7 [id=43c66cf1f4554c79b35bd2689050a945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.049705 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb563bd6802473fafedd08bb10d15c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.049741 747 tablet_service.cc:1467] Processing CreateTablet for tablet e6cc2c7b12f945ea8307a187e1553226 (DEFAULT_TABLE table=default.Table_18addc41a8074f018e26f238d2e7ca70 [id=e0737f359bfa4154b4fb71617efc829f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.050078 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6cc2c7b12f945ea8307a187e1553226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.038354 893 tablet_service.cc:1514] Processing DeleteTablet for tablet bd445a1adba34529aa3f7a44b991b177 with delete_type TABLET_DATA_DELETED (replaced by tablet 5496b2f965064f7788ebd74c62d1240a) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.051654 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b77c8d6adbf44375aff2782a83643255. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.055940 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 46a9910f88574eac90db9e00138c8dda
I20250109 20:52:45.057250 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a9910f88574eac90db9e00138c8dda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.058122 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_18addc41a8074f018e26f238d2e7ca70" } new_table_name: "default.Table_bc43af02a045473cac4e0b21c2dfc318" modify_external_catalogs: true
I20250109 20:52:45.058530 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a9910f88574eac90db9e00138c8dda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.058534 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7c349e16f1b547c2b54536d168ff9e4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.058948 1176 tablet_bootstrap.cc:492] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:45.058898 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 (DEFAULT_TABLE table=default.Table_c2a802cfb927494f92e603090370e543 [id=dccf417b39fc4d2d87812e340ecd842e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:45.059005 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c349e16f1b547c2b54536d168ff9e4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.059068 1176 ts_tablet_manager.cc:1397] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.089s user 0.019s sys 0.015s
I20250109 20:52:45.059298 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fb563bd6802473fafedd08bb10d15c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.060075 1176 raft_consensus.cc:357] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.060294 1176 raft_consensus.cc:383] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.060578 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_609a0ae37cfd4b6d8939b4829beb84d6" } modify_external_catalogs: true
I20250109 20:52:45.060752 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_609a0ae37cfd4b6d8939b4829beb84d6" } modify_external_catalogs: true
I20250109 20:52:45.060842 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a9910f88574eac90db9e00138c8dda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.062599 1176 raft_consensus.cc:738] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.062562 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_38382dc912fa4a28864b23129506ec12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.062976 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38382dc912fa4a28864b23129506ec12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.063894 1176 consensus_queue.cc:260] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.065471 1176 ts_tablet_manager.cc:1428] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:45.065754 1176 tablet_bootstrap.cc:492] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:45.068933 1176 tablet_bootstrap.cc:654] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.074699 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b5ba73bbb08a4db0adfc9e72a84a22e8
W20250109 20:52:45.076996 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c112bab5c5a42efab7582d83ca7eac1 consensus state has no leader
I20250109 20:52:45.077101 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b683408175de45c0bf7d7f7e14e15447" } modify_external_catalogs: true
I20250109 20:52:45.077244 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b683408175de45c0bf7d7f7e14e15447" } modify_external_catalogs: true
I20250109 20:52:45.077731 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 97205d09de264adb9d19cb32546b0942
I20250109 20:52:45.078053 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.078907 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_82246d90d1534a7e845106bde0e381f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.079356 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82246d90d1534a7e845106bde0e381f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.080822 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 97205d09de264adb9d19cb32546b0942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.081580 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0c3c6e0557ae46a0a8854d04f3d303e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.082031 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c3c6e0557ae46a0a8854d04f3d303e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.084676 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ce8c7068ce5540f5b0649980256146e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.085093 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce8c7068ce5540f5b0649980256146e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.093708 1166 ts_tablet_manager.cc:1918] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.095045 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_67d85bf32e4c4a9283bf48b7265d32cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.095517 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67d85bf32e4c4a9283bf48b7265d32cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:45.097632 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6cc2c7b12f945ea8307a187e1553226 consensus state has no leader
I20250109 20:52:45.098030 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cba0ffb1b394431ba30918911cdee7b7
I20250109 20:52:45.098721 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.099087 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 97205d09de264adb9d19cb32546b0942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.099910 1166 log.cc:1198] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c6a3dbc983534d65994fc0ef85d0dd2a
I20250109 20:52:45.100304 1166 ts_tablet_manager.cc:1939] T c6a3dbc983534d65994fc0ef85d0dd2a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:45.100843 891 tablet_service.cc:1514] Processing DeleteTablet for tablet cba0ffb1b394431ba30918911cdee7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.101315 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet c6a3dbc983534d65994fc0ef85d0dd2a (table default.Table_9785d67f4597409cb3bb0414d0166b4c [id=d1bd5b484fbf40388a3a22838448be2d]) successfully deleted
I20250109 20:52:45.101130 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet cba0ffb1b394431ba30918911cdee7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.104023 1165 tablet_bootstrap.cc:654] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.105688 1176 tablet_bootstrap.cc:492] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:45.105881 1176 ts_tablet_manager.cc:1397] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.007s
I20250109 20:52:45.108799 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 97205d09de264adb9d19cb32546b0942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.112771 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.119998 1331 tablet.cc:1721] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:45.135015 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b815e4bdccb14b8a992a4412bdff99d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.135273 449 master_service.cc:946] ReplaceTablet: received request to replace tablet f0e8ee4b86a04199865acd5f69cda384 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:45.135588 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b815e4bdccb14b8a992a4412bdff99d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.136659 1157 ts_tablet_manager.cc:1939] T 89fb6aeedc77452f99c46f03d6d8d089 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.136801 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.137235 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.137665 1158 tablet_bootstrap.cc:492] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:45.138319 1158 ts_tablet_manager.cc:1397] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.302s user 0.007s sys 0.008s
I20250109 20:52:45.138356 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_332207f19edb4562826601b0dbe41dce" } new_table_name: "default.Table_05dab35e3c154db288ec73033557c566" modify_external_catalogs: true
I20250109 20:52:45.139046 1158 raft_consensus.cc:357] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.139228 1158 raft_consensus.cc:383] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.139297 1158 raft_consensus.cc:738] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.139472 753 tablet_service.cc:1514] Processing DeleteTablet for tablet cba0ffb1b394431ba30918911cdee7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.141891 1157 tablet_replica.cc:331] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.145226 1157 raft_consensus.cc:2238] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.107092 1176 raft_consensus.cc:357] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.146345 1176 raft_consensus.cc:383] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.146456 1176 raft_consensus.cc:738] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.146672 1157 raft_consensus.cc:2267] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.147994 1157 ts_tablet_manager.cc:1905] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.101263 1166 tablet_replica.cc:331] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:45.149231 1166 raft_consensus.cc:2238] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.151841 1160 ts_tablet_manager.cc:1918] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.152360 1166 raft_consensus.cc:2267] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.154435 1187 raft_consensus.cc:491] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.154624 1187 raft_consensus.cc:513] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.154902 1330 tablet.cc:1721] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:45.155365 1187 leader_election.cc:290] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:45.155884 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660e60f368cb4e25b12ab3d7b2e314b1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:45.156122 777 raft_consensus.cc:2463] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:45.156543 990 leader_election.cc:304] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:45.156785 1187 raft_consensus.cc:2798] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:45.156850 1187 raft_consensus.cc:491] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:45.156939 1187 raft_consensus.cc:3054] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:45.157374 1166 ts_tablet_manager.cc:1905] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.152593 1333 tablet.cc:1721] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:45.155951 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660e60f368cb4e25b12ab3d7b2e314b1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:45.161710 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 660e60f368cb4e25b12ab3d7b2e314b1 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.162138 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 08ddedf78c0442daa9b1d05334f12b4d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.162185 918 raft_consensus.cc:2463] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:45.162483 453 master_service.cc:946] ReplaceTablet: received request to replace tablet c643c1ca41e048fdab95a067f899342d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.162977 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f0e8ee4b86a04199865acd5f69cda384
I20250109 20:52:45.161299 1160 log.cc:1198] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/d2aacfc7eea3490dbeb9d5c82d48766d
I20250109 20:52:45.163424 1160 ts_tablet_manager.cc:1939] T d2aacfc7eea3490dbeb9d5c82d48766d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.163662 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 89fb6aeedc77452f99c46f03d6d8d089 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.164444 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet d2aacfc7eea3490dbeb9d5c82d48766d (table default.Table_8c0e3ac68686429dadf4fcc158cf285b [id=0acf665b047a4f1dac232ef98359b372]) successfully deleted
I20250109 20:52:45.164633 890 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e8ee4b86a04199865acd5f69cda384 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.166461 753 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e8ee4b86a04199865acd5f69cda384 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.166699 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.172286 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.173029 1160 tablet_replica.cc:331] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.173254 1160 raft_consensus.cc:2238] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.173447 1160 raft_consensus.cc:2267] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.173990 1160 ts_tablet_manager.cc:1905] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:45.175570 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c112bab5c5a42efab7582d83ca7eac1 consensus state has no leader
I20250109 20:52:45.181741 1165 tablet_bootstrap.cc:492] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:45.181916 1165 ts_tablet_manager.cc:1397] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.138s user 0.005s sys 0.002s
I20250109 20:52:45.182623 1165 raft_consensus.cc:357] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.182853 1165 raft_consensus.cc:383] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.182934 1165 raft_consensus.cc:738] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.183220 1165 consensus_queue.cc:260] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.183807 1165 ts_tablet_manager.cc:1428] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:45.184345 1165 tablet_bootstrap.cc:492] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.186081 1165 tablet_bootstrap.cc:654] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.186902 899 tablet_service.cc:1467] Processing CreateTablet for tablet 06e615ce8e124cfe9432778b6b58e117 (DEFAULT_TABLE table=default.Table_759654c8879645ee84625859a22e72b2 [id=0256b60a9d99439b894214ef4f704f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.187237 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e615ce8e124cfe9432778b6b58e117. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:45.187453 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 (table default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet ded8e3a447c84b9b8ec4ccd70199ccd5 consensus state has no leader
I20250109 20:52:45.188966 891 tablet_service.cc:1467] Processing CreateTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 (DEFAULT_TABLE table=default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.189296 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c367a8ca184dfa9e4b11094ba76140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.194600 893 tablet_service.cc:1467] Processing CreateTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 (DEFAULT_TABLE table=default.Table_58060dd58dce4f7a8afa4d8c891c3ee3 [id=9a08f00bc4e74fc5861932c6f7cbbc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.194948 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.195825 1157 ts_tablet_manager.cc:1918] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.195946 1157 log.cc:1198] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/eeedf2e359c94a96ba6a518cefbb1fbe
I20250109 20:52:45.196193 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c643c1ca41e048fdab95a067f899342d
I20250109 20:52:45.196272 1157 ts_tablet_manager.cc:1939] T eeedf2e359c94a96ba6a518cefbb1fbe P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.196501 753 tablet_service.cc:1467] Processing CreateTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c (DEFAULT_TABLE table=default.Table_56c8a1eb8409437c856c4af9ece36c61 [id=fd0bcd24e74342f299f6b060b684af79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.196880 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbde2eecdaf4470dbc74b0c59f31643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.197053 895 tablet_service.cc:1514] Processing DeleteTablet for tablet c643c1ca41e048fdab95a067f899342d with delete_type TABLET_DATA_DELETED (replaced by tablet ea5d462230734767bb010f520c86c5ab) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.197335 453 catalog_manager.cc:6596] ReplaceTablet: tablet c643c1ca41e048fdab95a067f899342d deleted and replaced by tablet ea5d462230734767bb010f520c86c5ab
I20250109 20:52:45.197480 1157 tablet_replica.cc:331] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.197695 1157 raft_consensus.cc:2238] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.197887 1157 raft_consensus.cc:2267] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.198386 1157 ts_tablet_manager.cc:1905] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.201390 1160 ts_tablet_manager.cc:1918] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.201426 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet eeedf2e359c94a96ba6a518cefbb1fbe (table default.Table_e1e2df6604b3481583f4036b3aee752e [id=ae10da31c7cf49789ddf4c05c169c105]) successfully deleted
I20250109 20:52:45.202157 1160 log.cc:1198] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ba365ba04f6848328a9b14dc4c92ac2d
I20250109 20:52:45.202469 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 08ddedf78c0442daa9b1d05334f12b4d
I20250109 20:52:45.202906 446 catalog_manager.cc:6596] ReplaceTablet: tablet 08ddedf78c0442daa9b1d05334f12b4d deleted and replaced by tablet 7b7dc6809d6547f6b0354b64d7149bf4
I20250109 20:52:45.202924 1160 ts_tablet_manager.cc:1939] T ba365ba04f6848328a9b14dc4c92ac2d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.203481 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b7dc6809d6547f6b0354b64d7149bf4) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.203634 896 tablet_service.cc:1467] Processing CreateTablet for tablet fbcadec0c3184feaad4a883e5588f310 (DEFAULT_TABLE table=default.Table_e951676901de4adab1fb93a2d265da4f [id=97659ab8e1774179b7854d0985954b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.204037 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcadec0c3184feaad4a883e5588f310. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.207909 895 tablet_service.cc:1467] Processing CreateTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.208285 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a5815d9ef54e3e959f151ed08ae812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.209692 897 tablet_service.cc:1467] Processing CreateTablet for tablet 402280f75e51463a865c0fb521d0fd8a (DEFAULT_TABLE table=default.Table_e740022397a8451e830a159fa021e461 [id=24e621d6ed7e4a78a53808f4a8c2dd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.210033 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 402280f75e51463a865c0fb521d0fd8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.210875 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 660e60f368cb4e25b12ab3d7b2e314b1
I20250109 20:52:45.211341 451 catalog_manager.cc:6596] ReplaceTablet: tablet 660e60f368cb4e25b12ab3d7b2e314b1 deleted and replaced by tablet 2d96a569e6be4e01a26918d115cfbcf9
I20250109 20:52:45.212103 755 tablet_service.cc:1467] Processing CreateTablet for tablet 402280f75e51463a865c0fb521d0fd8a (DEFAULT_TABLE table=default.Table_e740022397a8451e830a159fa021e461 [id=24e621d6ed7e4a78a53808f4a8c2dd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.212451 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 402280f75e51463a865c0fb521d0fd8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.214337 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1] alter complete (version 4)
I20250109 20:52:45.215739 898 tablet_service.cc:1467] Processing CreateTablet for tablet b31b9785ee21418ba13b426ba94cfa9b (DEFAULT_TABLE table=default.Table_609b1c68b73f4e99ad6afa14ee14ad50 [id=4d18944cc25a4da6a21e3040370599e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.215808 892 tablet_service.cc:1467] Processing CreateTablet for tablet d8654c154dab465d8991910698027c3f (DEFAULT_TABLE table=default.Table_651b2a8068904467a07c19ff16909b5d [id=0b97ce1a3218466ca224e51d62b85577]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.216475 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b31b9785ee21418ba13b426ba94cfa9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.216857 748 tablet_service.cc:1467] Processing CreateTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 (DEFAULT_TABLE table=default.Table_b646ab4659864a2d9f3707100706389a [id=ccd17bbcb44f49a3b2eae73c6ace6c4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.217041 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 660e60f368cb4e25b12ab3d7b2e314b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d96a569e6be4e01a26918d115cfbcf9) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.217442 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8654c154dab465d8991910698027c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.217778 889 tablet_service.cc:1514] Processing DeleteTablet for tablet eeedf2e359c94a96ba6a518cefbb1fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:42 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.218480 758 tablet_service.cc:1467] Processing CreateTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 (DEFAULT_TABLE table=default.Table_58060dd58dce4f7a8afa4d8c891c3ee3 [id=9a08f00bc4e74fc5861932c6f7cbbc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.219069 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 660e60f368cb4e25b12ab3d7b2e314b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d96a569e6be4e01a26918d115cfbcf9) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.220803 891 tablet_service.cc:1467] Processing CreateTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 (DEFAULT_TABLE table=default.Table_b646ab4659864a2d9f3707100706389a [id=ccd17bbcb44f49a3b2eae73c6ace6c4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.221063 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b7dc6809d6547f6b0354b64d7149bf4) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.221279 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d4cad7594346dfafdee8e4c49d74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.222340 890 tablet_service.cc:1467] Processing CreateTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c (DEFAULT_TABLE table=default.Table_56c8a1eb8409437c856c4af9ece36c61 [id=fd0bcd24e74342f299f6b060b684af79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.222728 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbde2eecdaf4470dbc74b0c59f31643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.226389 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_651b2a8068904467a07c19ff16909b5d" } new_table_name: "default.Table_ea3048d16571434f9b5a83dadb1160be" modify_external_catalogs: true
I20250109 20:52:45.227811 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_834dbde1ccad4d8bb60efe0ae6ecf469" } new_table_name: "default.Table_0000587081d14ef1920a813cf3c7edaa" modify_external_catalogs: true
I20250109 20:52:45.233023 747 tablet_service.cc:1467] Processing CreateTablet for tablet b31b9785ee21418ba13b426ba94cfa9b (DEFAULT_TABLE table=default.Table_609b1c68b73f4e99ad6afa14ee14ad50 [id=4d18944cc25a4da6a21e3040370599e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.233458 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b31b9785ee21418ba13b426ba94cfa9b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:45.234828 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a89a0b47d10414d9b6a9123af1f4ab3 consensus state has no leader
W20250109 20:52:45.235258 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8654c154dab465d8991910698027c3f consensus state has no leader
I20250109 20:52:45.157639 1158 consensus_queue.cc:260] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.159164 1187 raft_consensus.cc:513] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.154281 1176 consensus_queue.cc:260] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.238456 1176 ts_tablet_manager.cc:1428] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.132s user 0.001s sys 0.002s
I20250109 20:52:45.238561 1187 leader_election.cc:290] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:45.166282 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e8ee4b86a04199865acd5f69cda384 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.238860 1176 tablet_bootstrap.cc:492] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:45.188892 1026 tablet_service.cc:1467] Processing CreateTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c (DEFAULT_TABLE table=default.Table_56c8a1eb8409437c856c4af9ece36c61 [id=fd0bcd24e74342f299f6b060b684af79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.239598 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbde2eecdaf4470dbc74b0c59f31643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.239789 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660e60f368cb4e25b12ab3d7b2e314b1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:45.192672 1017 tablet_service.cc:1467] Processing CreateTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 (DEFAULT_TABLE table=default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.239967 777 raft_consensus.cc:3054] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:45.240219 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c367a8ca184dfa9e4b11094ba76140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.240485 1176 tablet_bootstrap.cc:654] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.242339 777 raft_consensus.cc:2463] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:45.242763 990 leader_election.cc:304] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:45.243028 1240 raft_consensus.cc:2798] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:45.243182 1240 raft_consensus.cc:695] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:45.243470 1240 consensus_queue.cc:237] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.201910 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet c643c1ca41e048fdab95a067f899342d with delete_type TABLET_DATA_DELETED (replaced by tablet ea5d462230734767bb010f520c86c5ab) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.203272 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b7dc6809d6547f6b0354b64d7149bf4) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.208470 1027 tablet_service.cc:1467] Processing CreateTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 (DEFAULT_TABLE table=default.Table_58060dd58dce4f7a8afa4d8c891c3ee3 [id=9a08f00bc4e74fc5861932c6f7cbbc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.208849 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 (DEFAULT_TABLE table=default.Table_b646ab4659864a2d9f3707100706389a [id=ccd17bbcb44f49a3b2eae73c6ace6c4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.209208 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.209512 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 402280f75e51463a865c0fb521d0fd8a (DEFAULT_TABLE table=default.Table_e740022397a8451e830a159fa021e461 [id=24e621d6ed7e4a78a53808f4a8c2dd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.245659 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d4cad7594346dfafdee8e4c49d74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.210225 1025 tablet_service.cc:1467] Processing CreateTablet for tablet d8654c154dab465d8991910698027c3f (DEFAULT_TABLE table=default.Table_651b2a8068904467a07c19ff16909b5d [id=0b97ce1a3218466ca224e51d62b85577]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.211006 1019 tablet_service.cc:1467] Processing CreateTablet for tablet b31b9785ee21418ba13b426ba94cfa9b (DEFAULT_TABLE table=default.Table_609b1c68b73f4e99ad6afa14ee14ad50 [id=4d18944cc25a4da6a21e3040370599e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.217947 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 660e60f368cb4e25b12ab3d7b2e314b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d96a569e6be4e01a26918d115cfbcf9) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.246984 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.247443 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660e60f368cb4e25b12ab3d7b2e314b1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:45.247654 918 raft_consensus.cc:3054] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:45.248150 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ba365ba04f6848328a9b14dc4c92ac2d (table default.Table_2e5f440a1e9a4e6083d0857464311f92 [id=613376bfcfe64a468bcc9dc242929efe]) successfully deleted
I20250109 20:52:45.249384 1160 tablet_replica.cc:331] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.250119 1160 raft_consensus.cc:2238] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.250293 1160 raft_consensus.cc:2267] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.250609 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 402280f75e51463a865c0fb521d0fd8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.251300 918 raft_consensus.cc:2463] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:45.218921 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 06e615ce8e124cfe9432778b6b58e117 (DEFAULT_TABLE table=default.Table_759654c8879645ee84625859a22e72b2 [id=0256b60a9d99439b894214ef4f704f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.252597 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e615ce8e124cfe9432778b6b58e117. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.252864 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.253885 1160 ts_tablet_manager.cc:1905] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.254438 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1925a82eb711409ab2c5429c61edf9d8) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.256302 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a5815d9ef54e3e959f151ed08ae812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.257560 1157 ts_tablet_manager.cc:1918] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.257659 1157 log.cc:1198] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ab00e387fbc843b2841bef2d3289df34
I20250109 20:52:45.258013 1157 ts_tablet_manager.cc:1939] T ab00e387fbc843b2841bef2d3289df34 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.260192 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1e48be0e18b145708a9b6fa28ae21112" } modify_external_catalogs: true
I20250109 20:52:45.260385 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1e48be0e18b145708a9b6fa28ae21112" } modify_external_catalogs: true
I20250109 20:52:45.260848 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46d4cad7594346dfafdee8e4c49d74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.262917 1187 raft_consensus.cc:491] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.263094 1187 raft_consensus.cc:513] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.263870 1187 leader_election.cc:290] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.264863 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b31b9785ee21418ba13b426ba94cfa9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.264942 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97205d09de264adb9d19cb32546b0942" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:45.265479 989 leader_election.cc:343] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.266139 753 tablet_service.cc:1467] Processing CreateTablet for tablet e64207928ef141118e7fc8fe7bb2379b (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.266598 756 tablet_service.cc:1467] Processing CreateTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.267259 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8654c154dab465d8991910698027c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.269444 744 tablet_service.cc:1467] Processing CreateTablet for tablet d8654c154dab465d8991910698027c3f (DEFAULT_TABLE table=default.Table_651b2a8068904467a07c19ff16909b5d [id=0b97ce1a3218466ca224e51d62b85577]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.270004 899 tablet_service.cc:1467] Processing CreateTablet for tablet e64207928ef141118e7fc8fe7bb2379b (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.270380 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64207928ef141118e7fc8fe7bb2379b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.271484 1160 ts_tablet_manager.cc:1918] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.271595 1160 log.cc:1198] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/89fb6aeedc77452f99c46f03d6d8d089
I20250109 20:52:45.271958 1160 ts_tablet_manager.cc:1939] T 89fb6aeedc77452f99c46f03d6d8d089 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.273195 743 tablet_service.cc:1467] Processing CreateTablet for tablet 06e615ce8e124cfe9432778b6b58e117 (DEFAULT_TABLE table=default.Table_759654c8879645ee84625859a22e72b2 [id=0256b60a9d99439b894214ef4f704f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.273792 742 tablet_service.cc:1467] Processing CreateTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 (DEFAULT_TABLE table=default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.221954 1034 tablet_service.cc:1467] Processing CreateTablet for tablet fbcadec0c3184feaad4a883e5588f310 (DEFAULT_TABLE table=default.Table_e951676901de4adab1fb93a2d265da4f [id=97659ab8e1774179b7854d0985954b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.229070 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e64207928ef141118e7fc8fe7bb2379b (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.274448 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64207928ef141118e7fc8fe7bb2379b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.275986 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8f9dfe1265284477bd928e1cc0c9e477"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.276522 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f9dfe1265284477bd928e1cc0c9e477 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.278600 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2acc983c759e437cbce5b1a3a5e88cfe" } modify_external_catalogs: true
W20250109 20:52:45.278717 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1925a82eb711409ab2c5429c61edf9d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1925a82eb711409ab2c5429c61edf9d8 already in progress: creating tablet
I20250109 20:52:45.278793 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2acc983c759e437cbce5b1a3a5e88cfe" } modify_external_catalogs: true
I20250109 20:52:45.279318 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bece8c2a3b4c4f42981b775b7b6ec654" } new_table_name: "default.Table_0e564cfa1eb74fd8849dc79656731710" modify_external_catalogs: true
W20250109 20:52:45.279830 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 03b47ee787ca4dc5a4a56ae582d15b8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03b47ee787ca4dc5a4a56ae582d15b8b already in progress: creating tablet
I20250109 20:52:45.280454 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2cb493640d614e088e638e580754f9d4" } modify_external_catalogs: true
I20250109 20:52:45.280577 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2cb493640d614e088e638e580754f9d4" } modify_external_catalogs: true
W20250109 20:52:45.280733 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 32eb4dfcf96c4a05a55082e90ae04467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32eb4dfcf96c4a05a55082e90ae04467 already in progress: creating tablet
W20250109 20:52:45.281559 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2b9feb9de9dc4d22bafc0f7707943113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9feb9de9dc4d22bafc0f7707943113 already in progress: creating tablet
W20250109 20:52:45.282449 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ab593a602d61495eb7b323903ae610b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab593a602d61495eb7b323903ae610b6 already in progress: creating tablet
W20250109 20:52:45.283377 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 779153d4174b4887b5dc0bb355cb4a9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 779153d4174b4887b5dc0bb355cb4a9d already in progress: creating tablet
I20250109 20:52:45.284060 757 tablet_service.cc:1514] Processing DeleteTablet for tablet c643c1ca41e048fdab95a067f899342d with delete_type TABLET_DATA_DELETED (replaced by tablet ea5d462230734767bb010f520c86c5ab) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.284099 1160 tablet_replica.cc:331] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.284467 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97205d09de264adb9d19cb32546b0942" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:45.284690 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcadec0c3184feaad4a883e5588f310. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.284900 745 tablet_service.cc:1467] Processing CreateTablet for tablet fbcadec0c3184feaad4a883e5588f310 (DEFAULT_TABLE table=default.Table_e951676901de4adab1fb93a2d265da4f [id=97659ab8e1774179b7854d0985954b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.285014 1160 raft_consensus.cc:2238] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:45.285327 990 leader_election.cc:343] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.285462 990 leader_election.cc:304] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:45.286012 1157 tablet_replica.cc:331] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.286113 1187 raft_consensus.cc:2743] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.286218 1157 raft_consensus.cc:2238] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.286257 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ab00e387fbc843b2841bef2d3289df34 (table default.Table_49a9931b8903487db0856c10d60fc1e9 [id=8e13eabc06da4067aa8108f30c8d323d]) successfully deleted
I20250109 20:52:45.286375 1157 raft_consensus.cc:2267] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.286886 1157 ts_tablet_manager.cc:1905] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.292439 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a5815d9ef54e3e959f151ed08ae812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.293061 1157 ts_tablet_manager.cc:1918] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.293201 1157 log.cc:1198] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/68aa4a05c36e44b0aa6ee03780d2a815
I20250109 20:52:45.294299 1160 raft_consensus.cc:2267] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.293601 1157 ts_tablet_manager.cc:1939] T 68aa4a05c36e44b0aa6ee03780d2a815 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.295589 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 68aa4a05c36e44b0aa6ee03780d2a815 (table default.Table_27fb907c2b28435db3522088b3d352fa [id=4e6d2e7acdc84a8fa2d3bd56e8437974]) successfully deleted
I20250109 20:52:45.296468 1160 ts_tablet_manager.cc:1905] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.298611 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6d5f32a08cf941b1a9edf10a342e2395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.299767 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d5f32a08cf941b1a9edf10a342e2395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:45.300066 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 03b47ee787ca4dc5a4a56ae582d15b8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03b47ee787ca4dc5a4a56ae582d15b8b already in progress: creating tablet
I20250109 20:52:45.301236 1176 tablet_bootstrap.cc:492] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:45.301364 1176 ts_tablet_manager.cc:1397] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.015s
I20250109 20:52:45.301704 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5d1e5a64991f49a48a7b4286fc1ffe5f" } modify_external_catalogs: true
I20250109 20:52:45.301908 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5d1e5a64991f49a48a7b4286fc1ffe5f" } modify_external_catalogs: true
I20250109 20:52:45.302101 1176 raft_consensus.cc:357] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.302274 1176 raft_consensus.cc:383] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.302349 1176 raft_consensus.cc:738] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.302374 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e03d934cb25e4e96b3c9670d7698253a" } new_table_name: "default.Table_00a6d942e4674cf4b05f02518158c538" modify_external_catalogs: true
I20250109 20:52:45.302544 1176 consensus_queue.cc:260] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.303018 1176 ts_tablet_manager.cc:1428] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:45.303032 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64207928ef141118e7fc8fe7bb2379b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.303277 1176 tablet_bootstrap.cc:492] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:45.304009 1157 tablet_replica.cc:331] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.304196 1157 raft_consensus.cc:2238] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.304394 1157 raft_consensus.cc:2267] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.301826 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_532d28a5835d421ab3f13d74939ec028"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.304939 1157 ts_tablet_manager.cc:1905] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:45.305099 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_532d28a5835d421ab3f13d74939ec028 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.308156 1176 tablet_bootstrap.cc:654] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.309036 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5ba73bbb08a4db0adfc9e72a84a22e8) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.309980 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.315820 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8654c154dab465d8991910698027c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.316571 1158 ts_tablet_manager.cc:1428] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.178s user 0.000s sys 0.002s
I20250109 20:52:45.318017 1176 tablet_bootstrap.cc:492] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:45.318300 1176 ts_tablet_manager.cc:1397] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20250109 20:52:45.319017 1176 raft_consensus.cc:357] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.319201 1176 raft_consensus.cc:383] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.319262 1176 raft_consensus.cc:738] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.320513 1176 consensus_queue.cc:260] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.321530 1176 ts_tablet_manager.cc:1428] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:52:45.321583 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcadec0c3184feaad4a883e5588f310. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.321805 1176 tablet_bootstrap.cc:492] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:45.325791 1176 tablet_bootstrap.cc:654] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:45.327821 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35d7cead4f4149ae8de48eee2cfd8679 consensus state has no leader
I20250109 20:52:45.328356 1191 raft_consensus.cc:491] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.328531 1191 raft_consensus.cc:513] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.329212 1191 leader_election.cc:290] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:52:45.329746 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cba68160be6145ddbbd2019c9b5f0076 consensus state has no leader
W20250109 20:52:45.330531 854 leader_election.cc:343] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: eeedf2e359c94a96ba6a518cefbb1fbe
I20250109 20:52:45.336133 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:41583: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111),
I20250109 20:52:45.336282 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250109 20:52:45.336825 1240 raft_consensus.cc:491] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.336964 1240 raft_consensus.cc:513] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.337556 1240 leader_election.cc:290] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.338039 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e615ce8e124cfe9432778b6b58e117. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.338058 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d9969a46544d9954f71c77170e76c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:45.338240 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c367a8ca184dfa9e4b11094ba76140. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:45.338685 989 leader_election.cc:343] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:45.340947 854 leader_election.cc:343] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: eeedf2e359c94a96ba6a518cefbb1fbe
I20250109 20:52:45.341126 854 leader_election.cc:304] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:45.342103 1191 raft_consensus.cc:2743] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.347399 1165 tablet_bootstrap.cc:492] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:45.348526 1160 ts_tablet_manager.cc:1918] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.349174 1160 log.cc:1198] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ab00e387fbc843b2841bef2d3289df34
I20250109 20:52:45.349534 1160 ts_tablet_manager.cc:1939] T ab00e387fbc843b2841bef2d3289df34 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.350327 1160 tablet_replica.cc:331] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.350494 1160 raft_consensus.cc:2238] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.350663 1160 raft_consensus.cc:2267] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.350850 1165 ts_tablet_manager.cc:1397] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.167s user 0.012s sys 0.023s
I20250109 20:52:45.351289 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ab00e387fbc843b2841bef2d3289df34 (table default.Table_49a9931b8903487db0856c10d60fc1e9 [id=8e13eabc06da4067aa8108f30c8d323d]) successfully deleted
I20250109 20:52:45.351337 1160 ts_tablet_manager.cc:1905] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.351538 1165 raft_consensus.cc:357] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.351833 1165 raft_consensus.cc:383] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.351979 1165 raft_consensus.cc:738] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.352269 1165 consensus_queue.cc:260] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.357509 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cfa3005fdd40709a630d5bb80564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.359575 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d9969a46544d9954f71c77170e76c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:45.360684 990 leader_election.cc:343] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.360841 990 leader_election.cc:304] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:45.361272 1240 raft_consensus.cc:2743] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.362892 1165 ts_tablet_manager.cc:1428] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:52:45.365818 1240 raft_consensus.cc:491] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.365948 1240 raft_consensus.cc:513] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.366546 1240 leader_election.cc:290] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.369087 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc174d6bd9f44eaf9f66e54693597601" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:45.369612 989 leader_election.cc:343] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.369793 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2b4114526543461ebadfacba85114570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.370142 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_43ed51a629174e3bbaad39c5d8f43ef7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.370361 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b4114526543461ebadfacba85114570 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:45.370669 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43ed51a629174e3bbaad39c5d8f43ef7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.371814 1165 tablet_bootstrap.cc:492] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.372017 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4c9330ec3e904978bb8b1f0869a0c532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.372466 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c9330ec3e904978bb8b1f0869a0c532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.373631 1165 tablet_bootstrap.cc:654] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.364080 1160 ts_tablet_manager.cc:1918] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.374985 1160 log.cc:1198] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/bd445a1adba34529aa3f7a44b991b177
I20250109 20:52:45.375310 1160 ts_tablet_manager.cc:1939] T bd445a1adba34529aa3f7a44b991b177 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.375864 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet bd445a1adba34529aa3f7a44b991b177 (table default.Table_9af6ca0c3f0042358b13f0891ced6dea [id=16fc7304546a423abcbcd230d06efd59]) successfully deleted
I20250109 20:52:45.376076 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e951676901de4adab1fb93a2d265da4f" } modify_external_catalogs: true
I20250109 20:52:45.376240 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e951676901de4adab1fb93a2d265da4f" } modify_external_catalogs: true
I20250109 20:52:45.376731 1160 tablet_replica.cc:331] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
W20250109 20:52:45.376802 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 46a9910f88574eac90db9e00138c8dda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46a9910f88574eac90db9e00138c8dda already in progress: creating tablet
W20250109 20:52:45.377064 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5ba73bbb08a4db0adfc9e72a84a22e8 already in progress: creating tablet
W20250109 20:52:45.377287 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 97205d09de264adb9d19cb32546b0942 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97205d09de264adb9d19cb32546b0942 already in progress: creating tablet
W20250109 20:52:45.377504 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cba0ffb1b394431ba30918911cdee7b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cba0ffb1b394431ba30918911cdee7b7 already in progress: creating tablet
W20250109 20:52:45.377681 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f0e8ee4b86a04199865acd5f69cda384 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e8ee4b86a04199865acd5f69cda384 already in progress: creating tablet
W20250109 20:52:45.377900 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c643c1ca41e048fdab95a067f899342d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c643c1ca41e048fdab95a067f899342d already in progress: creating tablet
W20250109 20:52:45.379822 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 08ddedf78c0442daa9b1d05334f12b4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08ddedf78c0442daa9b1d05334f12b4d already in progress: creating tablet
I20250109 20:52:45.380026 1160 raft_consensus.cc:2238] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.380240 1160 raft_consensus.cc:2267] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.380353 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.380825 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.380890 1160 ts_tablet_manager.cc:1905] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.388795 1165 tablet_bootstrap.cc:492] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:45.389147 1165 ts_tablet_manager.cc:1397] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20250109 20:52:45.390204 1165 raft_consensus.cc:357] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.391644 1165 raft_consensus.cc:383] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.391995 1157 ts_tablet_manager.cc:1918] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.392164 1157 log.cc:1198] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/bd445a1adba34529aa3f7a44b991b177
I20250109 20:52:45.392150 1165 raft_consensus.cc:738] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.392536 1157 ts_tablet_manager.cc:1939] T bd445a1adba34529aa3f7a44b991b177 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.392807 1158 tablet_bootstrap.cc:492] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:45.392606 1165 consensus_queue.cc:260] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.394538 1158 tablet_bootstrap.cc:654] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.400831 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_53b6d02e81214c5ab40407b9d3fa6584"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.402684 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_08e4b27362104ad4af8816b4facabed7" } new_table_name: "default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4" modify_external_catalogs: true
W20250109 20:52:45.405309 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53b6d02e81214c5ab40407b9d3fa6584 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.402998 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet fbcadec0c3184feaad4a883e5588f310
I20250109 20:52:45.406700 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b9feb9de9dc4d22bafc0f7707943113 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.403501 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_af5240c88be241afba737b04efe6456e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.407634 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af5240c88be241afba737b04efe6456e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.407688 744 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcadec0c3184feaad4a883e5588f310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.408169 895 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcadec0c3184feaad4a883e5588f310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.404157 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bd35b5a0a8784eb98faf62ffffd690e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.412225 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd35b5a0a8784eb98faf62ffffd690e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.415094 1240 raft_consensus.cc:491] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.415259 1240 raft_consensus.cc:513] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.415832 1240 leader_election.cc:290] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.417076 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7ed9e8c921491c9d911d662f500451" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:45.419070 989 leader_election.cc:343] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:45.419657 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f672cbd6dffc49dbb69992ae7db979ea consensus state has no leader
I20250109 20:52:45.422497 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 4f5a90e65e80494a8b451f2bc6baa2b1 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.424873 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc174d6bd9f44eaf9f66e54693597601" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:45.425231 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7ed9e8c921491c9d911d662f500451" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:45.425742 990 leader_election.cc:343] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.425859 990 leader_election.cc:304] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:45.426252 990 leader_election.cc:343] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.426343 990 leader_election.cc:304] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:45.426589 1187 raft_consensus.cc:2743] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.426771 1187 raft_consensus.cc:2743] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.428751 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 6acadf60d566407aabc1a2395d1d235f from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.431025 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet bd445a1adba34529aa3f7a44b991b177 (table default.Table_9af6ca0c3f0042358b13f0891ced6dea [id=16fc7304546a423abcbcd230d06efd59]) successfully deleted
W20250109 20:52:45.432403 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1925a82eb711409ab2c5429c61edf9d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1925a82eb711409ab2c5429c61edf9d8 already in progress: creating tablet
W20250109 20:52:45.433020 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 779153d4174b4887b5dc0bb355cb4a9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 779153d4174b4887b5dc0bb355cb4a9d already in progress: creating tablet
I20250109 20:52:45.433135 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcadec0c3184feaad4a883e5588f310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:45.433728 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ab593a602d61495eb7b323903ae610b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab593a602d61495eb7b323903ae610b6 already in progress: creating tablet
W20250109 20:52:45.434298 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2b9feb9de9dc4d22bafc0f7707943113 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9feb9de9dc4d22bafc0f7707943113 already in progress: creating tablet
W20250109 20:52:45.434845 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 32eb4dfcf96c4a05a55082e90ae04467 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32eb4dfcf96c4a05a55082e90ae04467 already in progress: creating tablet
I20250109 20:52:45.435132 1157 tablet_replica.cc:331] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.435349 1157 raft_consensus.cc:2238] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:45.435717 1157 pending_rounds.cc:62] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Trying to abort 1 pending ops.
I20250109 20:52:45.435925 1157 pending_rounds.cc:69] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523631160119296 op_type: NO_OP noop_request { }
I20250109 20:52:45.436120 1157 raft_consensus.cc:2883] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:45.436293 1157 raft_consensus.cc:2267] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.437017 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_05dab35e3c154db288ec73033557c566" } new_table_name: "default.Table_5894addb4e7a40a8b557929948171f8b" modify_external_catalogs: true
I20250109 20:52:45.438107 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aa4a05c36e44b0aa6ee03780d2a815 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.440150 1166 ts_tablet_manager.cc:1918] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.440814 453 master_service.cc:946] ReplaceTablet: received request to replace tablet f1589b79a56148b6ab2946943de39fb2 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.441010 1166 log.cc:1198] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/03e06e64448a4fb2bf41fc6e3d0361a6
I20250109 20:52:45.441445 1166 ts_tablet_manager.cc:1939] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:45.442271 1165 ts_tablet_manager.cc:1428] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20250109 20:52:45.442286 1158 tablet_bootstrap.cc:492] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:45.442413 1158 ts_tablet_manager.cc:1397] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.002s
I20250109 20:52:45.443892 1158 raft_consensus.cc:357] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.444154 1158 raft_consensus.cc:383] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.444281 1158 raft_consensus.cc:738] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.444579 1158 consensus_queue.cc:260] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.444941 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7541b9bb11c44022ae972d09079fd75d" } new_table_name: "default.Table_b19d3251bcdf41eab27c97e89366d1c7" modify_external_catalogs: true
I20250109 20:52:45.445420 1157 ts_tablet_manager.cc:1905] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.448099 1165 tablet_bootstrap.cc:492] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.449059 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 38760b86fdfb4c8c8bcfd90874cbfd0d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.449851 1165 tablet_bootstrap.cc:654] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.444563 1160 ts_tablet_manager.cc:1918] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.450687 1160 log.cc:1198] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/660e60f368cb4e25b12ab3d7b2e314b1
I20250109 20:52:45.450903 1157 ts_tablet_manager.cc:1918] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:45.451020 1157 log.cc:1198] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/34cfa3005fdd40709a630d5bb80564db
I20250109 20:52:45.451175 1160 ts_tablet_manager.cc:1939] T 660e60f368cb4e25b12ab3d7b2e314b1 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.451401 1157 ts_tablet_manager.cc:1939] T 34cfa3005fdd40709a630d5bb80564db P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.452167 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 660e60f368cb4e25b12ab3d7b2e314b1 (table default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]) successfully deleted
I20250109 20:52:45.452452 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 34cfa3005fdd40709a630d5bb80564db (table default.Table_5c35bca9b68e42bb9075f51687f74d15 [id=261eb60785fc4d11909ad99fe7cb53d0]) successfully deleted
I20250109 20:52:45.452500 1160 tablet_replica.cc:331] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.452653 1160 raft_consensus.cc:2238] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.452898 1160 raft_consensus.cc:2267] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.453562 1160 ts_tablet_manager.cc:1905] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:45.454464 1296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:52:45.454865 1296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:52:45.455003 1296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250109 20:52:45.457526 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 46a9910f88574eac90db9e00138c8dda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46a9910f88574eac90db9e00138c8dda already in progress: creating tablet
I20250109 20:52:45.457852 1157 tablet_replica.cc:331] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:52:45.458055 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 97205d09de264adb9d19cb32546b0942 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97205d09de264adb9d19cb32546b0942 already in progress: creating tablet
I20250109 20:52:45.458077 1157 raft_consensus.cc:2238] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.458207 1157 raft_consensus.cc:2267] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:45.458240 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5ba73bbb08a4db0adfc9e72a84a22e8 already in progress: creating tablet
W20250109 20:52:45.458379 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cba0ffb1b394431ba30918911cdee7b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cba0ffb1b394431ba30918911cdee7b7 already in progress: creating tablet
W20250109 20:52:45.458549 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f0e8ee4b86a04199865acd5f69cda384 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e8ee4b86a04199865acd5f69cda384 already in progress: creating tablet
I20250109 20:52:45.458719 1157 ts_tablet_manager.cc:1905] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:45.461453 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c112bab5c5a42efab7582d83ca7eac1 consensus state has no leader
I20250109 20:52:45.463402 756 tablet_service.cc:1514] Processing DeleteTablet for tablet ab593a602d61495eb7b323903ae610b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.463507 1160 ts_tablet_manager.cc:1918] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.463888 1158 ts_tablet_manager.cc:1428] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20250109 20:52:45.464242 1158 tablet_bootstrap.cc:492] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:45.465273 1160 log.cc:1198] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/eeedf2e359c94a96ba6a518cefbb1fbe
I20250109 20:52:45.465847 1160 ts_tablet_manager.cc:1939] T eeedf2e359c94a96ba6a518cefbb1fbe P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:52:45.466507 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0463dd5de987418d8ea1b102b09137bf consensus state has no leader
I20250109 20:52:45.466895 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 38760b86fdfb4c8c8bcfd90874cbfd0d
I20250109 20:52:45.467343 451 catalog_manager.cc:6596] ReplaceTablet: tablet 38760b86fdfb4c8c8bcfd90874cbfd0d deleted and replaced by tablet 72a455033b674e5a94fc37d0275eb313
I20250109 20:52:45.467703 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d with delete_type TABLET_DATA_DELETED (replaced by tablet 72a455033b674e5a94fc37d0275eb313) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.468356 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet eeedf2e359c94a96ba6a518cefbb1fbe (table default.Table_e1e2df6604b3481583f4036b3aee752e [id=ae10da31c7cf49789ddf4c05c169c105]) successfully deleted
I20250109 20:52:45.469044 1160 tablet_replica.cc:331] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.469439 1160 raft_consensus.cc:2238] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.469839 1160 raft_consensus.cc:2267] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.470669 1166 tablet_replica.cc:331] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:45.470712 1160 ts_tablet_manager.cc:1905] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.472349 1166 raft_consensus.cc:2238] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.474334 1157 ts_tablet_manager.cc:1918] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.474609 1157 log.cc:1198] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/660e60f368cb4e25b12ab3d7b2e314b1
I20250109 20:52:45.474972 1157 ts_tablet_manager.cc:1939] T 660e60f368cb4e25b12ab3d7b2e314b1 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.476778 1160 ts_tablet_manager.cc:1918] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.477563 1157 tablet_replica.cc:331] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.477809 1157 raft_consensus.cc:2238] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.478042 1157 raft_consensus.cc:2267] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.478790 1157 ts_tablet_manager.cc:1905] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.479943 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 660e60f368cb4e25b12ab3d7b2e314b1 (table default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]) successfully deleted
W20250109 20:52:45.480206 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 08ddedf78c0442daa9b1d05334f12b4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08ddedf78c0442daa9b1d05334f12b4d already in progress: creating tablet
I20250109 20:52:45.481014 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d with delete_type TABLET_DATA_DELETED (replaced by tablet 72a455033b674e5a94fc37d0275eb313) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:45.481998 1296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:52:45.482210 1296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:52:45.482329 1296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:52:45.482453 1296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:52:45.482903 1160 log.cc:1198] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/34cfa3005fdd40709a630d5bb80564db
I20250109 20:52:45.483148 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 54d11b9148dc42bc91348e837af73972
I20250109 20:52:45.483886 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 11d303cd42a2428fa17f29b48c4f64f5
I20250109 20:52:45.483963 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d11b9148dc42bc91348e837af73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.484658 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.474634 1166 raft_consensus.cc:2267] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.485145 1160 ts_tablet_manager.cc:1939] T 34cfa3005fdd40709a630d5bb80564db P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.485522 1166 ts_tablet_manager.cc:1905] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.486498 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 34cfa3005fdd40709a630d5bb80564db (table default.Table_5c35bca9b68e42bb9075f51687f74d15 [id=261eb60785fc4d11909ad99fe7cb53d0]) successfully deleted
W20250109 20:52:45.487514 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet fbcadec0c3184feaad4a883e5588f310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcadec0c3184feaad4a883e5588f310 already in progress: creating tablet
I20250109 20:52:45.488071 1160 tablet_replica.cc:331] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.488653 1158 tablet_bootstrap.cc:654] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.488786 899 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8e7081cac440389bec5dfac2391beb (DEFAULT_TABLE table=default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3 [id=e1afc4c2550f49f4b3c7b17ea2f09f60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.489391 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8e7081cac440389bec5dfac2391beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.491129 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d11b9148dc42bc91348e837af73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.491294 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8e7081cac440389bec5dfac2391beb (DEFAULT_TABLE table=default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3 [id=e1afc4c2550f49f4b3c7b17ea2f09f60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.491345 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.491638 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8e7081cac440389bec5dfac2391beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.491708 742 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8e7081cac440389bec5dfac2391beb (DEFAULT_TABLE table=default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3 [id=e1afc4c2550f49f4b3c7b17ea2f09f60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.493527 895 tablet_service.cc:1467] Processing CreateTablet for tablet 773f2c6e6d7f415ab1218f2365c880a2 (DEFAULT_TABLE table=default.Table_67d85bf32e4c4a9283bf48b7265d32cc [id=f61e7aec1ea74c409648bee9f041ee0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.493881 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773f2c6e6d7f415ab1218f2365c880a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.494422 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7d981c7f13c3409f994bfdf0275e0116
I20250109 20:52:45.495158 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ea5d462230734767bb010f520c86c5ab
I20250109 20:52:45.495761 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f1589b79a56148b6ab2946943de39fb2
I20250109 20:52:45.496150 453 catalog_manager.cc:6596] ReplaceTablet: tablet f1589b79a56148b6ab2946943de39fb2 deleted and replaced by tablet b4b31bd4072941d5aaf8c43dada3ccc7
I20250109 20:52:45.496922 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (DEFAULT_TABLE table=default.Table_b815e4bdccb14b8a992a4412bdff99d8 [id=127a26e6b93f4873b9bab8bc378ac91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.497267 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.497574 745 tablet_service.cc:1467] Processing CreateTablet for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (DEFAULT_TABLE table=default.Table_b815e4bdccb14b8a992a4412bdff99d8 [id=127a26e6b93f4873b9bab8bc378ac91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.498009 743 tablet_service.cc:1467] Processing CreateTablet for tablet 773f2c6e6d7f415ab1218f2365c880a2 (DEFAULT_TABLE table=default.Table_67d85bf32e4c4a9283bf48b7265d32cc [id=f61e7aec1ea74c409648bee9f041ee0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.498325 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.498677 891 tablet_service.cc:1467] Processing CreateTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c (DEFAULT_TABLE table=default.Table_82246d90d1534a7e845106bde0e381f0 [id=a1ff467d633e4d999b85452ff6372219]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.498989 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95735385d5ab4fe0b29e33bad14eb95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.500617 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5d462230734767bb010f520c86c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.500870 897 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5d462230734767bb010f520c86c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.500909 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c (DEFAULT_TABLE table=default.Table_82246d90d1534a7e845106bde0e381f0 [id=a1ff467d633e4d999b85452ff6372219]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.501215 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2608f48f5950466e9ab2d12ee87503cb (DEFAULT_TABLE table=default.Table_d78dfd28e10e45549a210fad737f3627 [id=17d12fac83d6463fb91fab6729a08139]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.501299 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 5757b98100804c688f67e7111db2b622 (DEFAULT_TABLE table=default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.501396 1019 tablet_service.cc:1467] Processing CreateTablet for tablet deed945f2ecf4290ad002662c3f66030 (DEFAULT_TABLE table=default.Table_7c349e16f1b547c2b54536d168ff9e4a [id=339708d383f743e59fade9cec26192f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.501515 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2608f48f5950466e9ab2d12ee87503cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.501785 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 2608f48f5950466e9ab2d12ee87503cb (DEFAULT_TABLE table=default.Table_d78dfd28e10e45549a210fad737f3627 [id=17d12fac83d6463fb91fab6729a08139]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.501807 744 tablet_service.cc:1467] Processing CreateTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c (DEFAULT_TABLE table=default.Table_82246d90d1534a7e845106bde0e381f0 [id=a1ff467d633e4d999b85452ff6372219]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.502126 758 tablet_service.cc:1514] Processing DeleteTablet for tablet f1589b79a56148b6ab2946943de39fb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b31bd4072941d5aaf8c43dada3ccc7) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.502182 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db (DEFAULT_TABLE table=default.Table_0c3c6e0557ae46a0a8854d04f3d303e0 [id=c24699f5a3e742578b56eb8bf1a4dc97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.500622 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.502447 755 tablet_service.cc:1467] Processing CreateTablet for tablet 2608f48f5950466e9ab2d12ee87503cb (DEFAULT_TABLE table=default.Table_d78dfd28e10e45549a210fad737f3627 [id=17d12fac83d6463fb91fab6729a08139]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.500917 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f1589b79a56148b6ab2946943de39fb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b31bd4072941d5aaf8c43dada3ccc7) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.502126 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.502884 757 tablet_service.cc:1467] Processing CreateTablet for tablet 5757b98100804c688f67e7111db2b622 (DEFAULT_TABLE table=default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.502933 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deed945f2ecf4290ad002662c3f66030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.503106 749 tablet_service.cc:1467] Processing CreateTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db (DEFAULT_TABLE table=default.Table_0c3c6e0557ae46a0a8854d04f3d303e0 [id=c24699f5a3e742578b56eb8bf1a4dc97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.503319 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5d462230734767bb010f520c86c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.503393 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.503818 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2608f48f5950466e9ab2d12ee87503cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.504293 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95735385d5ab4fe0b29e33bad14eb95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.504439 896 tablet_service.cc:1514] Processing DeleteTablet for tablet f1589b79a56148b6ab2946943de39fb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b31bd4072941d5aaf8c43dada3ccc7) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.504599 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 38760b86fdfb4c8c8bcfd90874cbfd0d with delete_type TABLET_DATA_DELETED (replaced by tablet 72a455033b674e5a94fc37d0275eb313) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.443146 1176 tablet_bootstrap.cc:492] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:45.504777 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5757b98100804c688f67e7111db2b622. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.504806 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5757b98100804c688f67e7111db2b622 (DEFAULT_TABLE table=default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.504868 1176 ts_tablet_manager.cc:1397] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.183s user 0.002s sys 0.014s
I20250109 20:52:45.505086 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5757b98100804c688f67e7111db2b622. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.505757 1176 raft_consensus.cc:357] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.505970 1176 raft_consensus.cc:383] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.506063 1176 raft_consensus.cc:738] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.506309 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d11b9148dc42bc91348e837af73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.506500 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.506757 1176 consensus_queue.cc:260] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.507321 1176 ts_tablet_manager.cc:1428] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:45.507359 1166 ts_tablet_manager.cc:1918] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.508013 1166 log.cc:1198] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/175bf6c335a8405c97b296cf4718610b
I20250109 20:52:45.507977 888 tablet_service.cc:1467] Processing CreateTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db (DEFAULT_TABLE table=default.Table_0c3c6e0557ae46a0a8854d04f3d303e0 [id=c24699f5a3e742578b56eb8bf1a4dc97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.508272 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.509060 748 tablet_service.cc:1467] Processing CreateTablet for tablet deed945f2ecf4290ad002662c3f66030 (DEFAULT_TABLE table=default.Table_7c349e16f1b547c2b54536d168ff9e4a [id=339708d383f743e59fade9cec26192f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.509552 892 tablet_service.cc:1467] Processing CreateTablet for tablet deed945f2ecf4290ad002662c3f66030 (DEFAULT_TABLE table=default.Table_7c349e16f1b547c2b54536d168ff9e4a [id=339708d383f743e59fade9cec26192f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.509873 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deed945f2ecf4290ad002662c3f66030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.496930 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 773f2c6e6d7f415ab1218f2365c880a2 (DEFAULT_TABLE table=default.Table_67d85bf32e4c4a9283bf48b7265d32cc [id=f61e7aec1ea74c409648bee9f041ee0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.515451 756 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7dc6809d6547f6b0354b64d7149bf4 (DEFAULT_TABLE table=default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.516104 896 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 (DEFAULT_TABLE table=default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.516202 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8e7081cac440389bec5dfac2391beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.516359 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5757b98100804c688f67e7111db2b622. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.516403 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7539cf9ece4befaae6c521d471ea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.516654 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2608f48f5950466e9ab2d12ee87503cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.516026 753 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 (DEFAULT_TABLE table=default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.516861 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.516913 897 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7dc6809d6547f6b0354b64d7149bf4 (DEFAULT_TABLE table=default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.517024 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95735385d5ab4fe0b29e33bad14eb95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.517158 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773f2c6e6d7f415ab1218f2365c880a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.517423 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deed945f2ecf4290ad002662c3f66030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.523542 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b7dc6809d6547f6b0354b64d7149bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.524060 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773f2c6e6d7f415ab1218f2365c880a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.524885 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.531255 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b7dc6809d6547f6b0354b64d7149bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.532997 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d52320f2397e4526ae456263f71bcda2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.533634 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d52320f2397e4526ae456263f71bcda2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.534991 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7ad564a8e67d49bb8a46048a1c78ddd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.535394 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.538347 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_77d18f6149124e54b0a321ccd3bd8fab" } new_table_name: "default.Table_e8b35b282b6f4dc9bbb6038afd135892" modify_external_catalogs: true
I20250109 20:52:45.539966 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a35c15eced344d6fb5e7a4322a01a160"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.540367 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a35c15eced344d6fb5e7a4322a01a160 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.541941 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bd35b5a0a8784eb98faf62ffffd690e6" } new_table_name: "default.Table_d4b677aa33534a4d8261f52878181252" modify_external_catalogs: true
I20250109 20:52:45.544194 1157 ts_tablet_manager.cc:1918] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.545266 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 (DEFAULT_TABLE table=default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.546401 1157 log.cc:1198] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ded8e3a447c84b9b8ec4ccd70199ccd5
I20250109 20:52:45.546973 1157 ts_tablet_manager.cc:1939] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.548454 747 tablet_service.cc:1467] Processing CreateTablet for tablet ea5d462230734767bb010f520c86c5ab (DEFAULT_TABLE table=default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.550835 758 tablet_service.cc:1467] Processing CreateTablet for tablet 881ea20c17904d669c8a1fbd766ba559 (DEFAULT_TABLE table=default.Table_ce8c7068ce5540f5b0649980256146e3 [id=3e073742d9cf476387b0a0949059d933]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:45.556288 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3000b819a4cc4960909b98febd400631 consensus state has no leader
W20250109 20:52:45.560760 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbe8a9e5b5fe406b988126cd1859eabc has no consensus state
I20250109 20:52:45.561511 892 tablet_service.cc:1467] Processing CreateTablet for tablet 881ea20c17904d669c8a1fbd766ba559 (DEFAULT_TABLE table=default.Table_ce8c7068ce5540f5b0649980256146e3 [id=3e073742d9cf476387b0a0949059d933]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.561954 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881ea20c17904d669c8a1fbd766ba559. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.563694 1296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=46069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:52:45.565312 1176 tablet_bootstrap.cc:492] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:45.565346 893 tablet_service.cc:1467] Processing CreateTablet for tablet ea5d462230734767bb010f520c86c5ab (DEFAULT_TABLE table=default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.565840 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea5d462230734767bb010f520c86c5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.567040 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7539cf9ece4befaae6c521d471ea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.567813 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d96a569e6be4e01a26918d115cfbcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.568077 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881ea20c17904d669c8a1fbd766ba559. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.568202 890 tablet_service.cc:1467] Processing CreateTablet for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (DEFAULT_TABLE table=default.Table_b815e4bdccb14b8a992a4412bdff99d8 [id=127a26e6b93f4873b9bab8bc378ac91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.568557 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.568907 891 tablet_service.cc:1467] Processing CreateTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 (DEFAULT_TABLE table=default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.569401 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d96a569e6be4e01a26918d115cfbcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.575172 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7dc6809d6547f6b0354b64d7149bf4 (DEFAULT_TABLE table=default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.575558 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b7dc6809d6547f6b0354b64d7149bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.576323 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea5d462230734767bb010f520c86c5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.576433 1157 tablet_replica.cc:331] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.576644 1157 raft_consensus.cc:2238] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.576843 1157 raft_consensus.cc:2267] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.577450 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ded8e3a447c84b9b8ec4ccd70199ccd5 (table default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]) successfully deleted
W20250109 20:52:45.577853 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet fbcadec0c3184feaad4a883e5588f310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcadec0c3184feaad4a883e5588f310 already in progress: creating tablet
I20250109 20:52:45.581888 1025 tablet_service.cc:1467] Processing CreateTablet for tablet ea5d462230734767bb010f520c86c5ab (DEFAULT_TABLE table=default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.582206 1157 ts_tablet_manager.cc:1905] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.582338 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 (DEFAULT_TABLE table=default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.582571 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.587848 1160 raft_consensus.cc:2238] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.588135 1160 raft_consensus.cc:2267] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.588675 1165 tablet_bootstrap.cc:492] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:45.588914 1165 ts_tablet_manager.cc:1397] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.141s user 0.006s sys 0.011s
I20250109 20:52:45.588971 1160 ts_tablet_manager.cc:1905] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.582513 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 (DEFAULT_TABLE table=default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.583004 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea5d462230734767bb010f520c86c5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.589754 1165 raft_consensus.cc:357] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.589965 1165 raft_consensus.cc:383] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.590054 1165 raft_consensus.cc:738] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.590297 1165 consensus_queue.cc:260] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.590799 1157 ts_tablet_manager.cc:1918] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.590914 1157 log.cc:1198] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ab593a602d61495eb7b323903ae610b6
I20250109 20:52:45.591238 1157 ts_tablet_manager.cc:1939] T ab593a602d61495eb7b323903ae610b6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.586800 1187 raft_consensus.cc:491] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.592442 1187 raft_consensus.cc:513] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.593057 1187 leader_election.cc:290] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.594055 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f2f62aab954531ad6eb60b6f4661b8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:45.594480 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f2f62aab954531ad6eb60b6f4661b8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:45.594977 989 leader_election.cc:343] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:45.595294 990 leader_election.cc:343] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.595374 990 leader_election.cc:304] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:45.595947 1187 raft_consensus.cc:2743] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.596263 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d96a569e6be4e01a26918d115cfbcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.598985 1157 tablet_replica.cc:331] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.599130 1157 raft_consensus.cc:2238] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:45.599431 1157 raft_consensus.cc:2267] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.599932 1157 ts_tablet_manager.cc:1905] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:45.599917 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 -> Peer 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583): Couldn't send request to peer 36a29ab2002d4eb389cd11778988e1e9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250109 20:52:45.600113 893 tablet_service.cc:1514] Processing DeleteTablet for tablet ded8e3a447c84b9b8ec4ccd70199ccd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1925a82eb711409ab2c5429c61edf9d8) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.600371 1165 ts_tablet_manager.cc:1428] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250109 20:52:45.600579 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7539cf9ece4befaae6c521d471ea57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.601033 1165 tablet_bootstrap.cc:492] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.603506 1160 ts_tablet_manager.cc:1918] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.603632 1160 log.cc:1198] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/68aa4a05c36e44b0aa6ee03780d2a815
I20250109 20:52:45.603978 1160 ts_tablet_manager.cc:1939] T 68aa4a05c36e44b0aa6ee03780d2a815 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.588878 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 881ea20c17904d669c8a1fbd766ba559 (DEFAULT_TABLE table=default.Table_ce8c7068ce5540f5b0649980256146e3 [id=3e073742d9cf476387b0a0949059d933]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.604879 1160 tablet_replica.cc:331] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.604969 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 68aa4a05c36e44b0aa6ee03780d2a815 (table default.Table_27fb907c2b28435db3522088b3d352fa [id=4e6d2e7acdc84a8fa2d3bd56e8437974]) successfully deleted
I20250109 20:52:45.605135 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881ea20c17904d669c8a1fbd766ba559. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.606333 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ddea71a560eb417ba8159934da82d639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.606781 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddea71a560eb417ba8159934da82d639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.607252 1165 tablet_bootstrap.cc:654] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.605054 1160 raft_consensus.cc:2238] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.608182 1160 raft_consensus.cc:2267] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.608223 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7e276816fdaf4a70903bc13b0d7863fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.608671 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e276816fdaf4a70903bc13b0d7863fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.608803 1160 ts_tablet_manager.cc:1905] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.609722 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f739863bb39f41698e7e5a083fe352dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.610121 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f739863bb39f41698e7e5a083fe352dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.610913 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ab593a602d61495eb7b323903ae610b6 (table default.Table_ab5a0c56116047958eb48524522f7b27 [id=df6e0f6dba1942d9b232a020d5d2b4d9]) successfully deleted
W20250109 20:52:45.611227 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 11d303cd42a2428fa17f29b48c4f64f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11d303cd42a2428fa17f29b48c4f64f5 already in progress: creating tablet
I20250109 20:52:45.611297 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3" } modify_external_catalogs: true
W20250109 20:52:45.611449 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f1589b79a56148b6ab2946943de39fb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1589b79a56148b6ab2946943de39fb2 already in progress: creating tablet
I20250109 20:52:45.611507 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3" } modify_external_catalogs: true
I20250109 20:52:45.612057 1158 tablet_bootstrap.cc:492] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:45.612241 1158 ts_tablet_manager.cc:1397] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.148s user 0.008s sys 0.010s
I20250109 20:52:45.612993 1158 raft_consensus.cc:357] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.613576 1158 raft_consensus.cc:383] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.613723 1158 raft_consensus.cc:738] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.613828 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_68eba4e7c30d4954a84fbdf550b8cc07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.614418 1158 consensus_queue.cc:260] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
W20250109 20:52:45.616849 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68eba4e7c30d4954a84fbdf550b8cc07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.621543 1166 ts_tablet_manager.cc:1939] T 175bf6c335a8405c97b296cf4718610b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:45.622236 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7d981c7f13c3409f994bfdf0275e0116 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d981c7f13c3409f994bfdf0275e0116 already in progress: creating tablet
I20250109 20:52:45.622414 1176 tablet_bootstrap.cc:654] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.622606 451 master_service.cc:946] ReplaceTablet: received request to replace tablet e4c367a8ca184dfa9e4b11094ba76140 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:45.622915 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ea5d462230734767bb010f520c86c5ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea5d462230734767bb010f520c86c5ab already in progress: creating tablet
I20250109 20:52:45.623301 1165 tablet_bootstrap.cc:492] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:45.623534 1165 ts_tablet_manager.cc:1397] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20250109 20:52:45.623829 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d78dfd28e10e45549a210fad737f3627" } new_table_name: "default.Table_8a7392fff9c0401fb5c701ee92cf9c41" modify_external_catalogs: true
I20250109 20:52:45.623911 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_534eb61fef3c42c8bac87ac39a09ef45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.624409 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_534eb61fef3c42c8bac87ac39a09ef45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.624523 1166 tablet_replica.cc:331] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:45.624418 1165 raft_consensus.cc:357] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.624641 1165 raft_consensus.cc:383] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.624733 1165 raft_consensus.cc:738] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.624717 1166 raft_consensus.cc:2238] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.624905 1166 raft_consensus.cc:2267] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.624981 1165 consensus_queue.cc:260] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.625358 1166 ts_tablet_manager.cc:1905] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.625458 1165 ts_tablet_manager.cc:1428] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:45.625556 1157 ts_tablet_manager.cc:1918] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:45.625686 1157 log.cc:1198] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/38760b86fdfb4c8c8bcfd90874cbfd0d
I20250109 20:52:45.626071 1165 tablet_bootstrap.cc:492] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.626080 1157 ts_tablet_manager.cc:1939] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.627638 1165 tablet_bootstrap.cc:654] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:45.630582 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 46a9910f88574eac90db9e00138c8dda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46a9910f88574eac90db9e00138c8dda already in progress: creating tablet
W20250109 20:52:45.630846 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5ba73bbb08a4db0adfc9e72a84a22e8 already in progress: creating tablet
I20250109 20:52:45.631126 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]) successfully deleted
W20250109 20:52:45.631333 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 54d11b9148dc42bc91348e837af73972 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54d11b9148dc42bc91348e837af73972 already in progress: creating tablet
I20250109 20:52:45.637481 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 35d7cead4f4149ae8de48eee2cfd8679 (table default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]). Expected version 1 got 0
I20250109 20:52:45.638209 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3d586cebfe304d66a30047cfce9a2e16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.638650 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d586cebfe304d66a30047cfce9a2e16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.639936 1296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:52:45.640688 893 tablet_service.cc:1467] Processing CreateTablet for tablet be44ffc9e371452f8d5f4174fbfa752a (DEFAULT_TABLE table=default.Table_af5240c88be241afba737b04efe6456e [id=ee4103ad12894523a9f260bc6ff91419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.641098 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be44ffc9e371452f8d5f4174fbfa752a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.643085 1187 raft_consensus.cc:491] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.643321 1187 raft_consensus.cc:513] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.643977 1187 leader_election.cc:290] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.644676 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825efab5ff7b492ca3c7e3fe11ece098" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:45.644977 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4025cdef634d4626a33f01c362a1388a (DEFAULT_TABLE table=default.Table_d52320f2397e4526ae456263f71bcda2 [id=0b96e2cb7e234f498b8ef5e9a09e9a67]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:45.645224 990 leader_election.cc:343] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.645290 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4025cdef634d4626a33f01c362a1388a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.645675 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825efab5ff7b492ca3c7e3fe11ece098" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:45.646566 452 master_service.cc:946] ReplaceTablet: received request to replace tablet f5a5815d9ef54e3e959f151ed08ae812 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:45.646572 989 leader_election.cc:343] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.647171 989 leader_election.cc:304] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:45.647563 1187 raft_consensus.cc:2743] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.647524 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_536e03fce06b4511b8df3cdbc1603575"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.647997 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_536e03fce06b4511b8df3cdbc1603575 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.648976 891 tablet_service.cc:1467] Processing CreateTablet for tablet a58d439254f8418091e3f23c8c274240 (DEFAULT_TABLE table=default.Table_4c9330ec3e904978bb8b1f0869a0c532 [id=fbf8fa58eaee4147b61be651b770935b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.649327 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a58d439254f8418091e3f23c8c274240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.650820 1166 ts_tablet_manager.cc:1918] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.650949 1166 log.cc:1198] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/97205d09de264adb9d19cb32546b0942
I20250109 20:52:45.651376 1166 ts_tablet_manager.cc:1939] T 97205d09de264adb9d19cb32546b0942 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:45.652951 892 tablet_service.cc:1467] Processing CreateTablet for tablet b221e048f6f6463e84e630386c304a05 (DEFAULT_TABLE table=default.Table_43ed51a629174e3bbaad39c5d8f43ef7 [id=ce77658acdf34b7e90401b0982c045fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.653259 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b221e048f6f6463e84e630386c304a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.642112 1296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 20:52:45.656992 447 master_service.cc:946] ReplaceTablet: received request to replace tablet f7c277dcc04d4acdbc116e41ad41d7cf from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.658171 897 tablet_service.cc:1467] Processing CreateTablet for tablet a8da9781ab32429297d0c875c1971501 (DEFAULT_TABLE table=default.Table_2b4114526543461ebadfacba85114570 [id=8fddac7b2a084c15b76824d25240859d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.658550 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8da9781ab32429297d0c875c1971501. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.661631 1176 tablet_bootstrap.cc:492] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:45.661816 1176 ts_tablet_manager.cc:1397] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.154s user 0.004s sys 0.008s
I20250109 20:52:45.661931 899 tablet_service.cc:1467] Processing CreateTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 (DEFAULT_TABLE table=default.Table_532d28a5835d421ab3f13d74939ec028 [id=2a15fab6f69a4e399c87563e7e5be289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.662357 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4015c0e66dfd47bfb9855f82044f01e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.662604 1176 raft_consensus.cc:357] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.662815 1176 raft_consensus.cc:383] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.662889 1176 raft_consensus.cc:738] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.663147 1176 consensus_queue.cc:260] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.663271 887 tablet_service.cc:1467] Processing CreateTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc (DEFAULT_TABLE table=default.Table_d4b677aa33534a4d8261f52878181252 [id=2f89d33acf544985bafb538d400c0d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.663630 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe8a9e5b5fe406b988126cd1859eabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.664994 896 tablet_service.cc:1467] Processing CreateTablet for tablet 62caa775ef1241f68c0323a2d228c8cd (DEFAULT_TABLE table=default.Table_8f9dfe1265284477bd928e1cc0c9e477 [id=9675dd7ce663428483bab06a077d9c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.665310 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62caa775ef1241f68c0323a2d228c8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.666116 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 97205d09de264adb9d19cb32546b0942 (table default.Table_1bca442744ee4e04a46973fb21434cae [id=45220eb0588a4c62838a951101e7e04f]) successfully deleted
I20250109 20:52:45.666700 1031 tablet_service.cc:1467] Processing CreateTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc (DEFAULT_TABLE table=default.Table_d4b677aa33534a4d8261f52878181252 [id=2f89d33acf544985bafb538d400c0d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.667034 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe8a9e5b5fe406b988126cd1859eabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.667070 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4025cdef634d4626a33f01c362a1388a (DEFAULT_TABLE table=default.Table_d52320f2397e4526ae456263f71bcda2 [id=0b96e2cb7e234f498b8ef5e9a09e9a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.667415 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4025cdef634d4626a33f01c362a1388a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.667959 888 tablet_service.cc:1467] Processing CreateTablet for tablet 72a455033b674e5a94fc37d0275eb313 (DEFAULT_TABLE table=default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.668037 1030 tablet_service.cc:1467] Processing CreateTablet for tablet b221e048f6f6463e84e630386c304a05 (DEFAULT_TABLE table=default.Table_43ed51a629174e3bbaad39c5d8f43ef7 [id=ce77658acdf34b7e90401b0982c045fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.668330 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a455033b674e5a94fc37d0275eb313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.668350 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b221e048f6f6463e84e630386c304a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.669342 885 tablet_service.cc:1467] Processing CreateTablet for tablet bd3d0bfb39db40758e82be511d033b4c (DEFAULT_TABLE table=default.Table_53b6d02e81214c5ab40407b9d3fa6584 [id=b49ad0fe0808440c817b00f7dab76667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.669693 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3d0bfb39db40758e82be511d033b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.672923 889 tablet_service.cc:1467] Processing CreateTablet for tablet ca2b022ad9274c0e9b67560389d8dbed (DEFAULT_TABLE table=default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb [id=5d0cc83d6e3e4da4845ec153a35cd890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.672950 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a8da9781ab32429297d0c875c1971501 (DEFAULT_TABLE table=default.Table_2b4114526543461ebadfacba85114570 [id=8fddac7b2a084c15b76824d25240859d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.673195 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8da9781ab32429297d0c875c1971501. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.673211 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2b022ad9274c0e9b67560389d8dbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.675172 753 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.675626 756 tablet_service.cc:1467] Processing CreateTablet for tablet be44ffc9e371452f8d5f4174fbfa752a (DEFAULT_TABLE table=default.Table_af5240c88be241afba737b04efe6456e [id=ee4103ad12894523a9f260bc6ff91419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.675931 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be44ffc9e371452f8d5f4174fbfa752a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.677008 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 (DEFAULT_TABLE table=default.Table_532d28a5835d421ab3f13d74939ec028 [id=2a15fab6f69a4e399c87563e7e5be289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.677053 1158 ts_tablet_manager.cc:1428] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.065s user 0.002s sys 0.000s
I20250109 20:52:45.677290 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4015c0e66dfd47bfb9855f82044f01e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.677423 895 tablet_service.cc:1467] Processing CreateTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 (DEFAULT_TABLE table=default.Table_6d5f32a08cf941b1a9edf10a342e2395 [id=6f85f27d20024666b822558de484bc6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.677600 1158 tablet_bootstrap.cc:492] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:45.677728 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7307ebdb0c794451a6d91d23d69fddc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.679257 1158 tablet_bootstrap.cc:654] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.680969 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 62caa775ef1241f68c0323a2d228c8cd (DEFAULT_TABLE table=default.Table_8f9dfe1265284477bd928e1cc0c9e477 [id=9675dd7ce663428483bab06a077d9c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.680979 745 tablet_service.cc:1467] Processing CreateTablet for tablet 4025cdef634d4626a33f01c362a1388a (DEFAULT_TABLE table=default.Table_d52320f2397e4526ae456263f71bcda2 [id=0b96e2cb7e234f498b8ef5e9a09e9a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.681080 898 tablet_service.cc:1467] Processing CreateTablet for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (DEFAULT_TABLE table=default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.681267 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4025cdef634d4626a33f01c362a1388a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.681267 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62caa775ef1241f68c0323a2d228c8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.681331 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b31bd4072941d5aaf8c43dada3ccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.684489 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 72a455033b674e5a94fc37d0275eb313 (DEFAULT_TABLE table=default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.684824 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a455033b674e5a94fc37d0275eb313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.684931 746 tablet_service.cc:1467] Processing CreateTablet for tablet a58d439254f8418091e3f23c8c274240 (DEFAULT_TABLE table=default.Table_4c9330ec3e904978bb8b1f0869a0c532 [id=fbf8fa58eaee4147b61be651b770935b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.685271 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a58d439254f8418091e3f23c8c274240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.688418 1018 tablet_service.cc:1467] Processing CreateTablet for tablet ca2b022ad9274c0e9b67560389d8dbed (DEFAULT_TABLE table=default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb [id=5d0cc83d6e3e4da4845ec153a35cd890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.688622 743 tablet_service.cc:1467] Processing CreateTablet for tablet b221e048f6f6463e84e630386c304a05 (DEFAULT_TABLE table=default.Table_43ed51a629174e3bbaad39c5d8f43ef7 [id=ce77658acdf34b7e90401b0982c045fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.688817 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2b022ad9274c0e9b67560389d8dbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.689081 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b221e048f6f6463e84e630386c304a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.692991 747 tablet_service.cc:1467] Processing CreateTablet for tablet a8da9781ab32429297d0c875c1971501 (DEFAULT_TABLE table=default.Table_2b4114526543461ebadfacba85114570 [id=8fddac7b2a084c15b76824d25240859d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.693296 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8da9781ab32429297d0c875c1971501. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.696470 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_47b199ac3a5945978c6af0a999fe64c1" } new_table_name: "default.Table_96663f02560b43c1bef9c2cad86ae6c8" modify_external_catalogs: true
I20250109 20:52:45.696645 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b646ab4659864a2d9f3707100706389a" } new_table_name: "default.Table_49b7725a1f7647cd9b136c55bd8ed301" modify_external_catalogs: true
I20250109 20:52:45.696960 758 tablet_service.cc:1467] Processing CreateTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 (DEFAULT_TABLE table=default.Table_532d28a5835d421ab3f13d74939ec028 [id=2a15fab6f69a4e399c87563e7e5be289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.697266 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4015c0e66dfd47bfb9855f82044f01e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.698155 1033 tablet_service.cc:1467] Processing CreateTablet for tablet bd3d0bfb39db40758e82be511d033b4c (DEFAULT_TABLE table=default.Table_53b6d02e81214c5ab40407b9d3fa6584 [id=b49ad0fe0808440c817b00f7dab76667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.698511 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3d0bfb39db40758e82be511d033b4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:45.704814 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2608f48f5950466e9ab2d12ee87503cb consensus state has no leader
I20250109 20:52:45.704804 1191 raft_consensus.cc:491] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.704945 1191 raft_consensus.cc:513] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.705011 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 (DEFAULT_TABLE table=default.Table_6d5f32a08cf941b1a9edf10a342e2395 [id=6f85f27d20024666b822558de484bc6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.705305 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7307ebdb0c794451a6d91d23d69fddc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.705510 1191 leader_election.cc:290] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:45.707046 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d8369c071899441a9b05f8910c9743c2" } modify_external_catalogs: true
I20250109 20:52:45.707265 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d8369c071899441a9b05f8910c9743c2" } modify_external_catalogs: true
I20250109 20:52:45.707926 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (DEFAULT_TABLE table=default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.708240 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b31bd4072941d5aaf8c43dada3ccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.709517 1017 tablet_service.cc:1467] Processing CreateTablet for tablet a58d439254f8418091e3f23c8c274240 (DEFAULT_TABLE table=default.Table_4c9330ec3e904978bb8b1f0869a0c532 [id=fbf8fa58eaee4147b61be651b770935b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.710024 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d7cead4f4149ae8de48eee2cfd8679" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:45.710232 1054 raft_consensus.cc:2463] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:45.710628 854 leader_election.cc:304] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:45.710942 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a58d439254f8418091e3f23c8c274240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.711006 1191 raft_consensus.cc:2798] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:45.711102 1191 raft_consensus.cc:491] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:45.711198 1191 raft_consensus.cc:3054] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:45.712549 1024 tablet_service.cc:1467] Processing CreateTablet for tablet be44ffc9e371452f8d5f4174fbfa752a (DEFAULT_TABLE table=default.Table_af5240c88be241afba737b04efe6456e [id=ee4103ad12894523a9f260bc6ff91419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.713824 1191 raft_consensus.cc:513] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.714987 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be44ffc9e371452f8d5f4174fbfa752a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.714865 1191 leader_election.cc:290] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:45.715971 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d7cead4f4149ae8de48eee2cfd8679" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:45.716145 1054 raft_consensus.cc:3054] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:45.729375 1165 tablet_bootstrap.cc:492] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:45.729601 1165 ts_tablet_manager.cc:1397] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.004s
I20250109 20:52:45.730412 1165 raft_consensus.cc:357] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.730710 1165 raft_consensus.cc:383] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.730834 1165 raft_consensus.cc:738] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.731114 1165 consensus_queue.cc:260] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.733981 1165 ts_tablet_manager.cc:1428] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:45.734665 1165 tablet_bootstrap.cc:492] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.736600 1165 tablet_bootstrap.cc:654] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.746941 744 tablet_service.cc:1467] Processing CreateTablet for tablet ca2b022ad9274c0e9b67560389d8dbed (DEFAULT_TABLE table=default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb [id=5d0cc83d6e3e4da4845ec153a35cd890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.747468 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2b022ad9274c0e9b67560389d8dbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.751223 1268 raft_consensus.cc:491] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.751439 752 tablet_service.cc:1467] Processing CreateTablet for tablet bd3d0bfb39db40758e82be511d033b4c (DEFAULT_TABLE table=default.Table_53b6d02e81214c5ab40407b9d3fa6584 [id=b49ad0fe0808440c817b00f7dab76667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.752516 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 (DEFAULT_TABLE table=default.Table_6d5f32a08cf941b1a9edf10a342e2395 [id=6f85f27d20024666b822558de484bc6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.753751 742 tablet_service.cc:1467] Processing CreateTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc (DEFAULT_TABLE table=default.Table_d4b677aa33534a4d8261f52878181252 [id=2f89d33acf544985bafb538d400c0d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.753343 748 tablet_service.cc:1467] Processing CreateTablet for tablet 62caa775ef1241f68c0323a2d228c8cd (DEFAULT_TABLE table=default.Table_8f9dfe1265284477bd928e1cc0c9e477 [id=9675dd7ce663428483bab06a077d9c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.758455 755 tablet_service.cc:1467] Processing CreateTablet for tablet 72a455033b674e5a94fc37d0275eb313 (DEFAULT_TABLE table=default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.760288 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3d0bfb39db40758e82be511d033b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.760489 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7307ebdb0c794451a6d91d23d69fddc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.761410 1054 raft_consensus.cc:2463] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:45.761935 854 leader_election.cc:304] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:45.762239 1191 raft_consensus.cc:2798] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:45.764761 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.764863 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe8a9e5b5fe406b988126cd1859eabc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.764832 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d7cead4f4149ae8de48eee2cfd8679" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:45.765051 776 raft_consensus.cc:2463] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:45.658569 1160 ts_tablet_manager.cc:1918] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:45.766852 1160 log.cc:1198] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/38760b86fdfb4c8c8bcfd90874cbfd0d
I20250109 20:52:45.767325 1160 ts_tablet_manager.cc:1939] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.768504 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]) successfully deleted
W20250109 20:52:45.769246 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 54d11b9148dc42bc91348e837af73972 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54d11b9148dc42bc91348e837af73972 already in progress: creating tablet
W20250109 20:52:45.769879 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 11d303cd42a2428fa17f29b48c4f64f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11d303cd42a2428fa17f29b48c4f64f5 already in progress: creating tablet
W20250109 20:52:45.770496 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7d981c7f13c3409f994bfdf0275e0116 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d981c7f13c3409f994bfdf0275e0116 already in progress: creating tablet
W20250109 20:52:45.771103 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ea5d462230734767bb010f520c86c5ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea5d462230734767bb010f520c86c5ab already in progress: creating tablet
I20250109 20:52:45.771323 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4d8e7081cac440389bec5dfac2391beb
W20250109 20:52:45.771775 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f1589b79a56148b6ab2946943de39fb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1589b79a56148b6ab2946943de39fb2 already in progress: creating tablet
I20250109 20:52:45.772085 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_532d28a5835d421ab3f13d74939ec028" } modify_external_catalogs: true
I20250109 20:52:45.772234 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8e7081cac440389bec5dfac2391beb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.772254 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_532d28a5835d421ab3f13d74939ec028" } modify_external_catalogs: true
I20250109 20:52:45.772716 1160 tablet_replica.cc:331] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.772804 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e4c367a8ca184dfa9e4b11094ba76140
I20250109 20:52:45.772899 1160 raft_consensus.cc:2238] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.773077 1160 raft_consensus.cc:2267] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.773262 451 catalog_manager.cc:6596] ReplaceTablet: tablet e4c367a8ca184dfa9e4b11094ba76140 deleted and replaced by tablet 6619b3dce3ed4325a4871c3d40b42c9d
I20250109 20:52:45.773806 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f5a5815d9ef54e3e959f151ed08ae812
I20250109 20:52:45.773976 1160 ts_tablet_manager.cc:1905] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.774232 452 catalog_manager.cc:6596] ReplaceTablet: tablet f5a5815d9ef54e3e959f151ed08ae812 deleted and replaced by tablet ed5ea5d588834f1b8b2b057ec324aadc
I20250109 20:52:45.775025 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 35d7cead4f4149ae8de48eee2cfd8679 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.777887 890 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 with delete_type TABLET_DATA_DELETED (replaced by tablet 6619b3dce3ed4325a4871c3d40b42c9d) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.779008 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5285d784ef3043f397f8b3f5c0facc07" } modify_external_catalogs: true
I20250109 20:52:45.779187 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5285d784ef3043f397f8b3f5c0facc07" } modify_external_catalogs: true
I20250109 20:52:45.780637 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dc41b4efb2f245f29da9d86c1d4aef95" } modify_external_catalogs: true
I20250109 20:52:45.780864 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dc41b4efb2f245f29da9d86c1d4aef95" } modify_external_catalogs: true
I20250109 20:52:45.781008 1191 raft_consensus.cc:695] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Running, Role: LEADER
I20250109 20:52:45.781391 891 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 with delete_type TABLET_DATA_DELETED (replaced by tablet ed5ea5d588834f1b8b2b057ec324aadc) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.781457 1191 consensus_queue.cc:237] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.784072 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f7c277dcc04d4acdbc116e41ad41d7cf
I20250109 20:52:45.784511 447 catalog_manager.cc:6596] ReplaceTablet: tablet f7c277dcc04d4acdbc116e41ad41d7cf deleted and replaced by tablet 71ac8d15d48c49289712ce27768dc27b
W20250109 20:52:45.784951 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c40e5019321c497b814a281e0ab1a7f7 consensus state has no leader
I20250109 20:52:45.785298 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f73f4c073c5a4d26b61421f75c8da4fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.785717 1160 ts_tablet_manager.cc:1918] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:45.785785 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f73f4c073c5a4d26b61421f75c8da4fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.785826 1160 log.cc:1198] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ded8e3a447c84b9b8ec4ccd70199ccd5
I20250109 20:52:45.786115 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8e7081cac440389bec5dfac2391beb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.786183 1160 ts_tablet_manager.cc:1939] T ded8e3a447c84b9b8ec4ccd70199ccd5 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.786332 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 with delete_type TABLET_DATA_DELETED (replaced by tablet 6619b3dce3ed4325a4871c3d40b42c9d) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.786943 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 with delete_type TABLET_DATA_DELETED (replaced by tablet ed5ea5d588834f1b8b2b057ec324aadc) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.787118 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 607a78e281594e9a92c6aa63ee7bdf8d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:45.787451 1176 ts_tablet_manager.cc:1428] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.126s user 0.001s sys 0.001s
I20250109 20:52:45.788942 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ded8e3a447c84b9b8ec4ccd70199ccd5 (table default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]) successfully deleted
W20250109 20:52:45.789176 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4d8e7081cac440389bec5dfac2391beb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d8e7081cac440389bec5dfac2391beb already in progress: creating tablet
W20250109 20:52:45.789384 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e4c367a8ca184dfa9e4b11094ba76140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4c367a8ca184dfa9e4b11094ba76140 already in progress: creating tablet
W20250109 20:52:45.789597 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f5a5815d9ef54e3e959f151ed08ae812 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5a5815d9ef54e3e959f151ed08ae812 already in progress: creating tablet
I20250109 20:52:45.790051 891 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf with delete_type TABLET_DATA_DELETED (replaced by tablet 71ac8d15d48c49289712ce27768dc27b) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:45.790597 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f7c277dcc04d4acdbc116e41ad41d7cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7c277dcc04d4acdbc116e41ad41d7cf already in progress: creating tablet
W20250109 20:52:45.790627 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46d4cad7594346dfafdee8e4c49d74e6 consensus state has no leader
I20250109 20:52:45.791070 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6d5f32a08cf941b1a9edf10a342e2395" } new_table_name: "default.Table_2e81ac284d7740eb9e743bb21d11a625" modify_external_catalogs: true
I20250109 20:52:45.793207 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_db3d6cbe91b54958b543da192a338eb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.793923 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db3d6cbe91b54958b543da192a338eb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.796998 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bf29c4a64f7b4d49af314ad6211982b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.797425 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf29c4a64f7b4d49af314ad6211982b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.797747 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ddea71a560eb417ba8159934da82d639" } new_table_name: "default.Table_1a734623b43b4fab8389a393281e3a3f" modify_external_catalogs: true
I20250109 20:52:45.799214 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.799407 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ffd6ce78438841daa663b6344f062019
I20250109 20:52:45.800263 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd6ce78438841daa663b6344f062019 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.800853 1160 tablet_replica.cc:331] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.801085 1160 raft_consensus.cc:2238] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.801307 1160 raft_consensus.cc:2267] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:45.801482 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7307ebdb0c794451a6d91d23d69fddc6 consensus state has no leader
I20250109 20:52:45.802069 1160 ts_tablet_manager.cc:1905] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.802892 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd6ce78438841daa663b6344f062019 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.803256 1176 tablet_bootstrap.cc:492] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:45.804796 757 tablet_service.cc:1467] Processing CreateTablet for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (DEFAULT_TABLE table=default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.805217 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62caa775ef1241f68c0323a2d228c8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.805395 1176 tablet_bootstrap.cc:654] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.803421 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf with delete_type TABLET_DATA_DELETED (replaced by tablet 71ac8d15d48c49289712ce27768dc27b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.806968 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b31bd4072941d5aaf8c43dada3ccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.808190 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a455033b674e5a94fc37d0275eb313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.811036 890 tablet_service.cc:1467] Processing CreateTablet for tablet 0b083e6be42f41ecbf23ee57ec17b0a5 (DEFAULT_TABLE table=default.Table_7e276816fdaf4a70903bc13b0d7863fe [id=68f5eeffe12b4cda81ebb874ada4dfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.811421 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b083e6be42f41ecbf23ee57ec17b0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.812633 891 tablet_service.cc:1467] Processing CreateTablet for tablet 66e4bfa188b545349f358adfd0bc714b (DEFAULT_TABLE table=default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.812891 1166 tablet_replica.cc:331] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:45.813006 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66e4bfa188b545349f358adfd0bc714b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.813164 1166 raft_consensus.cc:2238] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:45.813443 1166 pending_rounds.cc:62] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Trying to abort 1 pending ops.
I20250109 20:52:45.813513 1166 pending_rounds.cc:69] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523633639620608 op_type: NO_OP noop_request { }
I20250109 20:52:45.813624 1166 raft_consensus.cc:2883] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:45.813728 1166 raft_consensus.cc:2267] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.814512 1166 ts_tablet_manager.cc:1905] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.816973 899 tablet_service.cc:1467] Processing CreateTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc (DEFAULT_TABLE table=default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.817277 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705a7c04f3f948e1a5e486d5ccd48ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.819104 1160 ts_tablet_manager.cc:1918] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.819223 1160 log.cc:1198] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/c6a3dbc983534d65994fc0ef85d0dd2a
I20250109 20:52:45.819525 1160 ts_tablet_manager.cc:1939] T c6a3dbc983534d65994fc0ef85d0dd2a P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.820967 892 tablet_service.cc:1467] Processing CreateTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee (DEFAULT_TABLE table=default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.821589 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet c6a3dbc983534d65994fc0ef85d0dd2a (table default.Table_9785d67f4597409cb3bb0414d0166b4c [id=d1bd5b484fbf40388a3a22838448be2d]) successfully deleted
I20250109 20:52:45.822649 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b405d16cc62444f2ad94fb69bb9fb6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.823186 1176 tablet_bootstrap.cc:492] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:45.823335 1176 ts_tablet_manager.cc:1397] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250109 20:52:45.759171 1268 raft_consensus.cc:513] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
I20250109 20:52:45.824211 1176 raft_consensus.cc:357] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.824448 1176 raft_consensus.cc:383] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.824550 1176 raft_consensus.cc:738] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.824875 1176 consensus_queue.cc:260] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.828994 1157 tablet_replica.cc:331] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:52:45.829366 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ffd6ce78438841daa663b6344f062019 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffd6ce78438841daa663b6344f062019 already in progress: creating tablet
I20250109 20:52:45.829545 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 with delete_type TABLET_DATA_DELETED (replaced by tablet 6619b3dce3ed4325a4871c3d40b42c9d) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.828910 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d7cead4f4149ae8de48eee2cfd8679" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:45.833711 1176 ts_tablet_manager.cc:1428] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250109 20:52:45.833686 893 tablet_service.cc:1467] Processing CreateTablet for tablet af8c0bf27e994e6aa4cbedca16febe12 (DEFAULT_TABLE table=default.Table_ddea71a560eb417ba8159934da82d639 [id=3ecc13ed899846fe8cf09444c0c25504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.834723 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf with delete_type TABLET_DATA_DELETED (replaced by tablet 71ac8d15d48c49289712ce27768dc27b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.834888 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 with delete_type TABLET_DATA_DELETED (replaced by tablet ed5ea5d588834f1b8b2b057ec324aadc) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.835047 753 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd6ce78438841daa663b6344f062019 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.835564 1158 tablet_bootstrap.cc:492] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:45.835736 1158 ts_tablet_manager.cc:1397] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.158s user 0.024s sys 0.022s
I20250109 20:52:45.836158 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1630e4438e4337ba026b41f630a480 (DEFAULT_TABLE table=default.Table_a35c15eced344d6fb5e7a4322a01a160 [id=b78805ca41e647e1927d2564fb00b21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.836791 1157 raft_consensus.cc:2238] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.836947 1157 raft_consensus.cc:2267] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:45.837133 712 leader_election.cc:343] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Not found: Tablet not found: c6a3dbc983534d65994fc0ef85d0dd2a
I20250109 20:52:45.837479 777 raft_consensus.cc:3054] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:45.837968 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8e7081cac440389bec5dfac2391beb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.838358 1268 leader_election.cc:290] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.839533 1157 ts_tablet_manager.cc:1905] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.839583 756 tablet_service.cc:1467] Processing CreateTablet for tablet 66e4bfa188b545349f358adfd0bc714b (DEFAULT_TABLE table=default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.840793 753 tablet_service.cc:1467] Processing CreateTablet for tablet 0b083e6be42f41ecbf23ee57ec17b0a5 (DEFAULT_TABLE table=default.Table_7e276816fdaf4a70903bc13b0d7863fe [id=68f5eeffe12b4cda81ebb874ada4dfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.841135 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b083e6be42f41ecbf23ee57ec17b0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.841455 1158 raft_consensus.cc:357] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.841688 1158 raft_consensus.cc:383] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.841778 1158 raft_consensus.cc:738] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:45.841974 713 leader_election.cc:343] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: c6a3dbc983534d65994fc0ef85d0dd2a
I20250109 20:52:45.842317 713 leader_election.cc:304] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:52:45.842494 713 raft_consensus.cc:2681] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250109 20:52:45.842582 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cba0ffb1b394431ba30918911cdee7b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cba0ffb1b394431ba30918911cdee7b7 already in progress: creating tablet
I20250109 20:52:45.843636 741 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1630e4438e4337ba026b41f630a480 (DEFAULT_TABLE table=default.Table_a35c15eced344d6fb5e7a4322a01a160 [id=b78805ca41e647e1927d2564fb00b21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.844403 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1630e4438e4337ba026b41f630a480. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:45.846311 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f0e8ee4b86a04199865acd5f69cda384 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e8ee4b86a04199865acd5f69cda384 already in progress: creating tablet
I20250109 20:52:45.847033 751 tablet_service.cc:1467] Processing CreateTablet for tablet af8c0bf27e994e6aa4cbedca16febe12 (DEFAULT_TABLE table=default.Table_ddea71a560eb417ba8159934da82d639 [id=3ecc13ed899846fe8cf09444c0c25504]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:45.847436 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 08ddedf78c0442daa9b1d05334f12b4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08ddedf78c0442daa9b1d05334f12b4d already in progress: creating tablet
I20250109 20:52:45.850140 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 35d7cead4f4149ae8de48eee2cfd8679
I20250109 20:52:45.850279 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 607a78e281594e9a92c6aa63ee7bdf8d
I20250109 20:52:45.850553 448 catalog_manager.cc:6596] ReplaceTablet: tablet 35d7cead4f4149ae8de48eee2cfd8679 deleted and replaced by tablet cd596614521a4a92a71b781b162f70b6
I20250109 20:52:45.850623 447 catalog_manager.cc:6596] ReplaceTablet: tablet 607a78e281594e9a92c6aa63ee7bdf8d deleted and replaced by tablet 3d8d67b624984a0bb327d045c55476da
I20250109 20:52:45.851543 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 35d7cead4f4149ae8de48eee2cfd8679 with delete_type TABLET_DATA_DELETED (replaced by tablet cd596614521a4a92a71b781b162f70b6) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:45.851588 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af8c0bf27e994e6aa4cbedca16febe12 consensus state has no leader
I20250109 20:52:45.851656 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 268d9969a46544d9954f71c77170e76c
I20250109 20:52:45.851910 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d with delete_type TABLET_DATA_DELETED (replaced by tablet 3d8d67b624984a0bb327d045c55476da) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.841660 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1630e4438e4337ba026b41f630a480 (DEFAULT_TABLE table=default.Table_a35c15eced344d6fb5e7a4322a01a160 [id=b78805ca41e647e1927d2564fb00b21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.852298 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4015c0e66dfd47bfb9855f82044f01e7
I20250109 20:52:45.852541 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 268d9969a46544d9954f71c77170e76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.852591 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1630e4438e4337ba026b41f630a480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.853011 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.842048 1031 tablet_service.cc:1467] Processing CreateTablet for tablet af8c0bf27e994e6aa4cbedca16febe12 (DEFAULT_TABLE table=default.Table_ddea71a560eb417ba8159934da82d639 [id=3ecc13ed899846fe8cf09444c0c25504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.853991 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6636fd244b2448a4a8de2eda78a535a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.854133 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 35d7cead4f4149ae8de48eee2cfd8679 with delete_type TABLET_DATA_DELETED (replaced by tablet cd596614521a4a92a71b781b162f70b6) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.842341 1033 tablet_service.cc:1467] Processing CreateTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee (DEFAULT_TABLE table=default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:45.854542 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6636fd244b2448a4a8de2eda78a535a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.842532 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.842808 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc (DEFAULT_TABLE table=default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.854996 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8c0bf27e994e6aa4cbedca16febe12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.843123 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 0b083e6be42f41ecbf23ee57ec17b0a5 (DEFAULT_TABLE table=default.Table_7e276816fdaf4a70903bc13b0d7863fe [id=68f5eeffe12b4cda81ebb874ada4dfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.844192 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 66e4bfa188b545349f358adfd0bc714b (DEFAULT_TABLE table=default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.855602 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.855762 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 71ac8d15d48c49289712ce27768dc27b
I20250109 20:52:45.855942 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 268d9969a46544d9954f71c77170e76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.856415 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8c0bf27e994e6aa4cbedca16febe12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.857110 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d with delete_type TABLET_DATA_DELETED (replaced by tablet 3d8d67b624984a0bb327d045c55476da) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:45.857312 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b405d16cc62444f2ad94fb69bb9fb6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.858248 750 tablet_service.cc:1467] Processing CreateTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc (DEFAULT_TABLE table=default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.858583 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705a7c04f3f948e1a5e486d5ccd48ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.859561 754 tablet_service.cc:1467] Processing CreateTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee (DEFAULT_TABLE table=default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.859877 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b405d16cc62444f2ad94fb69bb9fb6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.861182 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 705a7c04f3f948e1a5e486d5ccd48ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.862099 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3da7386e032a4516b7fa5ae4b2418db3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.862038 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c762911889354a9497c8d731f5e6c578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.862507 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3da7386e032a4516b7fa5ae4b2418db3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:45.862514 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c762911889354a9497c8d731f5e6c578 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.863504 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b083e6be42f41ecbf23ee57ec17b0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.863730 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e677fcf0892746659b8f6d9f5dce6bc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:45.864151 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e677fcf0892746659b8f6d9f5dce6bc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.865980 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66e4bfa188b545349f358adfd0bc714b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.867837 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 268d9969a46544d9954f71c77170e76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.868072 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.883966 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8c0bf27e994e6aa4cbedca16febe12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.888991 1160 tablet_replica.cc:331] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.889176 1160 raft_consensus.cc:2238] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:45.889590 1160 pending_rounds.cc:62] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Trying to abort 1 pending ops.
I20250109 20:52:45.889633 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1630e4438e4337ba026b41f630a480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.889660 1160 pending_rounds.cc:69] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523635847524352 op_type: NO_OP noop_request { }
I20250109 20:52:45.889806 1160 raft_consensus.cc:2883] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:45.889899 1160 raft_consensus.cc:2267] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.901793 777 raft_consensus.cc:2463] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:45.903821 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66e4bfa188b545349f358adfd0bc714b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.908160 1176 tablet_bootstrap.cc:492] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:45.908207 1352 consensus_queue.cc:1159] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "44d473955919435cb679edd5f8afe2c2" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250109 20:52:45.909524 1160 ts_tablet_manager.cc:1905] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.911901 1176 tablet_bootstrap.cc:654] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.911777 1158 consensus_queue.cc:260] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.912341 1158 ts_tablet_manager.cc:1428] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.077s user 0.001s sys 0.000s
I20250109 20:52:45.912621 1158 tablet_bootstrap.cc:492] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:45.914721 1158 tablet_bootstrap.cc:654] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.915097 1160 ts_tablet_manager.cc:1918] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:45.915212 1160 log.cc:1198] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/35d7cead4f4149ae8de48eee2cfd8679
I20250109 20:52:45.915524 1160 ts_tablet_manager.cc:1939] T 35d7cead4f4149ae8de48eee2cfd8679 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:45.915830 1187 raft_consensus.cc:491] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.916009 1187 raft_consensus.cc:513] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.916509 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 35d7cead4f4149ae8de48eee2cfd8679 (table default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
W20250109 20:52:45.917080 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 607a78e281594e9a92c6aa63ee7bdf8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607a78e281594e9a92c6aa63ee7bdf8d already in progress: creating tablet
I20250109 20:52:45.917388 1187 leader_election.cc:290] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:45.917629 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 268d9969a46544d9954f71c77170e76c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 268d9969a46544d9954f71c77170e76c already in progress: creating tablet
I20250109 20:52:45.917949 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5a90e65e80494a8b451f2bc6baa2b1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:45.918161 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4015c0e66dfd47bfb9855f82044f01e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4015c0e66dfd47bfb9855f82044f01e7 already in progress: creating tablet
I20250109 20:52:45.918159 777 raft_consensus.cc:2463] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:45.918551 990 leader_election.cc:304] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:45.918586 1157 ts_tablet_manager.cc:1918] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:45.918718 1157 log.cc:1198] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/c6a3dbc983534d65994fc0ef85d0dd2a
I20250109 20:52:45.918826 1187 raft_consensus.cc:2798] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:45.918926 1187 raft_consensus.cc:491] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:45.919067 1157 ts_tablet_manager.cc:1939] T c6a3dbc983534d65994fc0ef85d0dd2a P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:45.919071 1187 raft_consensus.cc:3054] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:45.921002 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5a90e65e80494a8b451f2bc6baa2b1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:45.921458 989 leader_election.cc:343] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.921990 1187 raft_consensus.cc:513] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.923275 1187 leader_election.cc:290] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.923552 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5a90e65e80494a8b451f2bc6baa2b1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:45.923795 777 raft_consensus.cc:3054] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:45.923871 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet c6a3dbc983534d65994fc0ef85d0dd2a (table default.Table_9785d67f4597409cb3bb0414d0166b4c [id=d1bd5b484fbf40388a3a22838448be2d]) successfully deleted
W20250109 20:52:45.924113 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e4c367a8ca184dfa9e4b11094ba76140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4c367a8ca184dfa9e4b11094ba76140 already in progress: creating tablet
W20250109 20:52:45.924381 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f5a5815d9ef54e3e959f151ed08ae812 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5a5815d9ef54e3e959f151ed08ae812 already in progress: creating tablet
W20250109 20:52:45.924573 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ffd6ce78438841daa663b6344f062019 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffd6ce78438841daa663b6344f062019 already in progress: creating tablet
W20250109 20:52:45.924762 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4d8e7081cac440389bec5dfac2391beb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d8e7081cac440389bec5dfac2391beb already in progress: creating tablet
I20250109 20:52:45.924897 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 35d7cead4f4149ae8de48eee2cfd8679 with delete_type TABLET_DATA_DELETED (replaced by tablet cd596614521a4a92a71b781b162f70b6) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:45.924970 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 268d9969a46544d9954f71c77170e76c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 268d9969a46544d9954f71c77170e76c already in progress: creating tablet
W20250109 20:52:45.925148 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4015c0e66dfd47bfb9855f82044f01e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4015c0e66dfd47bfb9855f82044f01e7 already in progress: creating tablet
I20250109 20:52:45.925892 1176 tablet_bootstrap.cc:492] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:45.926036 1176 ts_tablet_manager.cc:1397] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20250109 20:52:45.926456 1157 tablet_replica.cc:331] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:45.926627 1157 raft_consensus.cc:2238] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.927220 1176 raft_consensus.cc:357] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.927384 1176 raft_consensus.cc:383] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.927430 1176 raft_consensus.cc:738] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.926776 1157 raft_consensus.cc:2267] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:45.928068 1157 ts_tablet_manager.cc:1905] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.927960 1176 consensus_queue.cc:260] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:45.928850 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d with delete_type TABLET_DATA_DELETED (replaced by tablet 3d8d67b624984a0bb327d045c55476da) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.930725 1176 ts_tablet_manager.cc:1428] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250109 20:52:45.932355 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f5a90e65e80494a8b451f2bc6baa2b1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:45.932390 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_00a6d942e4674cf4b05f02518158c538" } new_table_name: "default.Table_b70135c5a9b548538e791703e28af8f7" modify_external_catalogs: true
W20250109 20:52:45.932842 989 leader_election.cc:343] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.936511 1176 tablet_bootstrap.cc:492] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:45.940449 1176 tablet_bootstrap.cc:654] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.943006 1165 tablet_bootstrap.cc:492] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:45.943197 1165 ts_tablet_manager.cc:1397] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.209s user 0.040s sys 0.067s
W20250109 20:52:45.944687 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cba68160be6145ddbbd2019c9b5f0076 consensus state has no leader
I20250109 20:52:45.945026 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 35d7cead4f4149ae8de48eee2cfd8679 with delete_type TABLET_DATA_DELETED (replaced by tablet cd596614521a4a92a71b781b162f70b6) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.945369 893 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a3dbc983534d65994fc0ef85d0dd2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.944669 1165 raft_consensus.cc:357] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
W20250109 20:52:45.945700 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 35d7cead4f4149ae8de48eee2cfd8679 because the tablet was not found. No further retry: Not found: Tablet not found: 35d7cead4f4149ae8de48eee2cfd8679
I20250109 20:52:45.945844 1165 raft_consensus.cc:383] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.945933 1165 raft_consensus.cc:738] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.946240 1165 consensus_queue.cc:260] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } }
W20250109 20:52:45.946465 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c6a3dbc983534d65994fc0ef85d0dd2a because the tablet was not found. No further retry: Not found: Tablet not found: c6a3dbc983534d65994fc0ef85d0dd2a
I20250109 20:52:45.946756 1165 ts_tablet_manager.cc:1428] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250109 20:52:45.947033 1165 tablet_bootstrap.cc:492] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.949041 885 tablet_service.cc:1467] Processing CreateTablet for tablet 91dbccae0af24a629600a92e7eadfee7 (DEFAULT_TABLE table=default.Table_534eb61fef3c42c8bac87ac39a09ef45 [id=0e350177d27e4c8b8bc3cede2166517e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.949247 1165 tablet_bootstrap.cc:654] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:45.949420 1187 raft_consensus.cc:491] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:45.949586 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91dbccae0af24a629600a92e7eadfee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.949599 1187 raft_consensus.cc:513] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.950224 1187 leader_election.cc:290] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:45.951100 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c6016d88b54eb390f880945ddb1ea6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:45.951483 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c6016d88b54eb390f880945ddb1ea6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:45.954625 989 leader_election.cc:343] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.954859 1165 tablet_bootstrap.cc:492] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
W20250109 20:52:45.954896 990 leader_election.cc:343] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:45.954988 1165 ts_tablet_manager.cc:1397] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20250109 20:52:45.954980 990 leader_election.cc:304] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:45.955384 1187 raft_consensus.cc:2743] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:45.955768 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ab593a602d61495eb7b323903ae610b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:45.955705 1165 raft_consensus.cc:357] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.955904 1165 raft_consensus.cc:383] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:45.955997 1165 raft_consensus.cc:738] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:45.956121 1160 tablet_replica.cc:331] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:45.956140 777 raft_consensus.cc:2463] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:45.956234 1166 ts_tablet_manager.cc:1918] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:45.956276 1160 raft_consensus.cc:2238] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:45.956328 1166 log.cc:1198] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/660e60f368cb4e25b12ab3d7b2e314b1
I20250109 20:52:45.956226 1165 consensus_queue.cc:260] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:45.956432 1160 raft_consensus.cc:2267] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:45.959101 1356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:45.959448 893 tablet_service.cc:1467] Processing CreateTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 (DEFAULT_TABLE table=default.Table_536e03fce06b4511b8df3cdbc1603575 [id=fc92038a07574bcf9e5c970ba84eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.956676 1165 ts_tablet_manager.cc:1428] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:45.961441 990 leader_election.cc:304] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:45.961884 1165 tablet_bootstrap.cc:492] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:45.962133 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5ba73bbb08a4db0adfc9e72a84a22e8) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.962039 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6e345f747be74c0c9cfb28cf6ff90e93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.962334 754 tablet_service.cc:1467] Processing CreateTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 (DEFAULT_TABLE table=default.Table_536e03fce06b4511b8df3cdbc1603575 [id=fc92038a07574bcf9e5c970ba84eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:45.962524 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e345f747be74c0c9cfb28cf6ff90e93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.962651 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f215eee55d4e6a97a384df6ff9d596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.965353 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f215eee55d4e6a97a384df6ff9d596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.966135 742 tablet_service.cc:1467] Processing CreateTablet for tablet 6619b3dce3ed4325a4871c3d40b42c9d (DEFAULT_TABLE table=default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.966396 753 tablet_service.cc:1467] Processing CreateTablet for tablet 91dbccae0af24a629600a92e7eadfee7 (DEFAULT_TABLE table=default.Table_534eb61fef3c42c8bac87ac39a09ef45 [id=0e350177d27e4c8b8bc3cede2166517e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.966693 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6619b3dce3ed4325a4871c3d40b42c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.967494 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91dbccae0af24a629600a92e7eadfee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.974273 1165 tablet_bootstrap.cc:654] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:45.977393 1360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:45.977586 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5e3e729549234c39a601e29da97094d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:45.978005 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_43ed51a629174e3bbaad39c5d8f43ef7" } modify_external_catalogs: true
I20250109 20:52:45.978183 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_43ed51a629174e3bbaad39c5d8f43ef7" } modify_external_catalogs: true
W20250109 20:52:45.978381 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e3e729549234c39a601e29da97094d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:45.980500 1160 ts_tablet_manager.cc:1905] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:45.986060 753 tablet_service.cc:1467] Processing CreateTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.986508 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5ea5d588834f1b8b2b057ec324aadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.987224 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1838fc762deb4f71b0fd91a2eaa397ea" } new_table_name: "default.Table_6f5675a262804b5b98db035a4c98ddcf" modify_external_catalogs: true
I20250109 20:52:45.989609 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b221e048f6f6463e84e630386c304a05
W20250109 20:52:45.993721 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d8d67b624984a0bb327d045c55476da has no consensus state
I20250109 20:52:45.994011 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b221e048f6f6463e84e630386c304a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:45.994202 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b815e4bdccb14b8a992a4412bdff99d8" } new_table_name: "default.Table_66bc870f54384b4080562e0d880ce055" modify_external_catalogs: true
I20250109 20:52:45.996320 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2ad301c11943b69b59be18d6775513 (DEFAULT_TABLE table=default.Table_f73f4c073c5a4d26b61421f75c8da4fe [id=3a1cd7e71eab4f90ae054b885cee1626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:45.996826 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2ad301c11943b69b59be18d6775513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:45.998816 1158 tablet_bootstrap.cc:492] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:45.999061 1158 ts_tablet_manager.cc:1397] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.086s user 0.010s sys 0.001s
W20250109 20:52:45.999619 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba consensus state has no leader
I20250109 20:52:46.000139 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 66e4bfa188b545349f358adfd0bc714b from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.000577 1158 raft_consensus.cc:357] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.001116 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8b85547ea1c642a693a94f23f77df03f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.001498 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_534eb61fef3c42c8bac87ac39a09ef45" } modify_external_catalogs: true
I20250109 20:52:46.001662 890 tablet_service.cc:1467] Processing CreateTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.001690 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_534eb61fef3c42c8bac87ac39a09ef45" } modify_external_catalogs: true
W20250109 20:52:46.001688 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b85547ea1c642a693a94f23f77df03f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.002362 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b221e048f6f6463e84e630386c304a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.002940 1158 raft_consensus.cc:383] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.003033 1158 raft_consensus.cc:738] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.003306 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d311b482665944949ba9d26a91d7fd39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.003788 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d311b482665944949ba9d26a91d7fd39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.004143 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2b4114526543461ebadfacba85114570" } new_table_name: "default.Table_4c30a9425b844ee187a29bc12cd2704e" modify_external_catalogs: true
W20250109 20:52:46.004277 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 45235.6us
I20250109 20:52:46.004838 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 (DEFAULT_TABLE table=default.Table_536e03fce06b4511b8df3cdbc1603575 [id=fc92038a07574bcf9e5c970ba84eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.003281 1158 consensus_queue.cc:260] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.005542 1158 ts_tablet_manager.cc:1428] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250109 20:52:46.005882 1158 tablet_bootstrap.cc:492] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:46.006126 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5ea5d588834f1b8b2b057ec324aadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.009881 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b221e048f6f6463e84e630386c304a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.010393 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2ad301c11943b69b59be18d6775513 (DEFAULT_TABLE table=default.Table_f73f4c073c5a4d26b61421f75c8da4fe [id=3a1cd7e71eab4f90ae054b885cee1626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.010778 742 tablet_service.cc:1467] Processing CreateTablet for tablet 47726b58d6714cd1b46848a16a34ff37 (DEFAULT_TABLE table=default.Table_3d586cebfe304d66a30047cfce9a2e16 [id=3123de0f3f7644739ee589a4c7e9edec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.010996 1187 raft_consensus.cc:2798] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:46.011154 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47726b58d6714cd1b46848a16a34ff37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.012224 1160 ts_tablet_manager.cc:1918] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.012352 1160 log.cc:1198] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ab593a602d61495eb7b323903ae610b6
I20250109 20:52:46.013563 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 175bf6c335a8405c97b296cf4718610b with delete_type TABLET_DATA_DELETED (replaced by tablet 11d303cd42a2428fa17f29b48c4f64f5) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.014081 891 tablet_service.cc:1467] Processing CreateTablet for tablet 4d2ad301c11943b69b59be18d6775513 (DEFAULT_TABLE table=default.Table_f73f4c073c5a4d26b61421f75c8da4fe [id=3a1cd7e71eab4f90ae054b885cee1626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.017086 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 66e4bfa188b545349f358adfd0bc714b
I20250109 20:52:46.017510 452 catalog_manager.cc:6596] ReplaceTablet: tablet 66e4bfa188b545349f358adfd0bc714b deleted and replaced by tablet a1ed00f7954143c9ba9bf65d8a8a68d2
I20250109 20:52:46.018205 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e4bfa188b545349f358adfd0bc714b with delete_type TABLET_DATA_DELETED (replaced by tablet a1ed00f7954143c9ba9bf65d8a8a68d2) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.018345 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2ad301c11943b69b59be18d6775513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.021849 899 tablet_service.cc:1467] Processing CreateTablet for tablet 47726b58d6714cd1b46848a16a34ff37 (DEFAULT_TABLE table=default.Table_3d586cebfe304d66a30047cfce9a2e16 [id=3123de0f3f7644739ee589a4c7e9edec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.022348 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47726b58d6714cd1b46848a16a34ff37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.026131 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f215eee55d4e6a97a384df6ff9d596. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.026383 893 tablet_service.cc:1467] Processing CreateTablet for tablet 6619b3dce3ed4325a4871c3d40b42c9d (DEFAULT_TABLE table=default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.026743 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6619b3dce3ed4325a4871c3d40b42c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.028031 1017 tablet_service.cc:1467] Processing CreateTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.028399 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5ea5d588834f1b8b2b057ec324aadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.029022 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 91dbccae0af24a629600a92e7eadfee7
W20250109 20:52:46.029831 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8da9781ab32429297d0c875c1971501 consensus state has no leader
I20250109 20:52:46.030912 1157 ts_tablet_manager.cc:1918] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.031064 1157 log.cc:1198] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/35d7cead4f4149ae8de48eee2cfd8679
I20250109 20:52:46.031440 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dbccae0af24a629600a92e7eadfee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.031523 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 47726b58d6714cd1b46848a16a34ff37 (DEFAULT_TABLE table=default.Table_3d586cebfe304d66a30047cfce9a2e16 [id=3123de0f3f7644739ee589a4c7e9edec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.031720 750 tablet_service.cc:1467] Processing CreateTablet for tablet 71ac8d15d48c49289712ce27768dc27b (DEFAULT_TABLE table=default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.031759 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 91dbccae0af24a629600a92e7eadfee7 (DEFAULT_TABLE table=default.Table_534eb61fef3c42c8bac87ac39a09ef45 [id=0e350177d27e4c8b8bc3cede2166517e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.032037 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91dbccae0af24a629600a92e7eadfee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.032080 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ac8d15d48c49289712ce27768dc27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.032519 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 71ac8d15d48c49289712ce27768dc27b (DEFAULT_TABLE table=default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.033241 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47726b58d6714cd1b46848a16a34ff37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.033386 1158 tablet_bootstrap.cc:654] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.031461 1157 ts_tablet_manager.cc:1939] T 35d7cead4f4149ae8de48eee2cfd8679 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:46.036010 1157 tablet_replica.cc:331] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:46.036188 1157 raft_consensus.cc:2238] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.036330 1157 raft_consensus.cc:2267] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.039225 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ac8d15d48c49289712ce27768dc27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.039316 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dbccae0af24a629600a92e7eadfee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.039930 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 35d7cead4f4149ae8de48eee2cfd8679 (table default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
W20250109 20:52:46.040194 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 607a78e281594e9a92c6aa63ee7bdf8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607a78e281594e9a92c6aa63ee7bdf8d already in progress: creating tablet
I20250109 20:52:46.044871 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dbccae0af24a629600a92e7eadfee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.045017 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6619b3dce3ed4325a4871c3d40b42c9d (DEFAULT_TABLE table=default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.045377 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6619b3dce3ed4325a4871c3d40b42c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.046469 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e4bfa188b545349f358adfd0bc714b with delete_type TABLET_DATA_DELETED (replaced by tablet a1ed00f7954143c9ba9bf65d8a8a68d2) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.047861 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2ad301c11943b69b59be18d6775513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.049531 1160 ts_tablet_manager.cc:1939] T ab593a602d61495eb7b323903ae610b6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:46.052161 1187 raft_consensus.cc:695] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:46.052413 1187 consensus_queue.cc:237] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.054257 892 tablet_service.cc:1467] Processing CreateTablet for tablet 71ac8d15d48c49289712ce27768dc27b (DEFAULT_TABLE table=default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.055413 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ac8d15d48c49289712ce27768dc27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.055406 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ab593a602d61495eb7b323903ae610b6 (table default.Table_ab5a0c56116047958eb48524522f7b27 [id=df6e0f6dba1942d9b232a020d5d2b4d9]) successfully deleted
W20250109 20:52:46.056249 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b221e048f6f6463e84e630386c304a05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b221e048f6f6463e84e630386c304a05 already in progress: creating tablet
W20250109 20:52:46.056788 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 91dbccae0af24a629600a92e7eadfee7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91dbccae0af24a629600a92e7eadfee7 already in progress: creating tablet
I20250109 20:52:46.059965 1157 ts_tablet_manager.cc:1905] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.061338 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 32eb4dfcf96c4a05a55082e90ae04467 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.061841 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 268d9969a46544d9954f71c77170e76c from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.062355 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_592b3f74ec6c48fdb0ac9daff1341dd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.062885 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_592b3f74ec6c48fdb0ac9daff1341dd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.063285 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_79f5ff4b60444c1fb9e227b39c89e4f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.063805 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79f5ff4b60444c1fb9e227b39c89e4f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.064132 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_536e03fce06b4511b8df3cdbc1603575" } new_table_name: "default.Table_8f9fc791c58d4d3fbd9ad92c0a88003f" modify_external_catalogs: true
I20250109 20:52:46.064664 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e4bfa188b545349f358adfd0bc714b with delete_type TABLET_DATA_DELETED (replaced by tablet a1ed00f7954143c9ba9bf65d8a8a68d2) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.065335 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b70135c5a9b548538e791703e28af8f7" } modify_external_catalogs: true
I20250109 20:52:46.065567 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b70135c5a9b548538e791703e28af8f7" } modify_external_catalogs: true
I20250109 20:52:46.069536 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4c9330ec3e904978bb8b1f0869a0c532" } modify_external_catalogs: true
I20250109 20:52:46.069706 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4c9330ec3e904978bb8b1f0869a0c532" } modify_external_catalogs: true
I20250109 20:52:46.070176 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_82246d90d1534a7e845106bde0e381f0" } modify_external_catalogs: true
I20250109 20:52:46.070333 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_82246d90d1534a7e845106bde0e381f0" } modify_external_catalogs: true
I20250109 20:52:46.073601 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044c212aada4df382d147c9a15ea4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.073699 1158 tablet_bootstrap.cc:492] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:46.073855 1158 ts_tablet_manager.cc:1397] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.001s
I20250109 20:52:46.074730 1158 raft_consensus.cc:357] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.075052 1158 raft_consensus.cc:383] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.075138 1158 raft_consensus.cc:738] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.075378 1158 consensus_queue.cc:260] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:46.076197 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34f215eee55d4e6a97a384df6ff9d596 consensus state has no leader
I20250109 20:52:46.077011 1166 ts_tablet_manager.cc:1939] T 660e60f368cb4e25b12ab3d7b2e314b1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.077791 1158 ts_tablet_manager.cc:1428] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250109 20:52:46.078400 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 66e4bfa188b545349f358adfd0bc714b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66e4bfa188b545349f358adfd0bc714b already in progress: creating tablet
I20250109 20:52:46.080811 1158 tablet_bootstrap.cc:492] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:46.083669 1158 tablet_bootstrap.cc:654] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.084424 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f7702e93eb894ea5bbf0af322f94c29c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.084996 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7702e93eb894ea5bbf0af322f94c29c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.086993 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 6a334724611e4303ba9d987033d872ba from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.087507 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_455cff8c7fb2449f908ca21e7da023af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.087908 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_455cff8c7fb2449f908ca21e7da023af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.088248 1157 ts_tablet_manager.cc:1918] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.089278 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 660e60f368cb4e25b12ab3d7b2e314b1 (table default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]) successfully deleted
I20250109 20:52:46.089725 1166 tablet_replica.cc:331] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:46.089960 1166 raft_consensus.cc:2238] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:46.090175 1166 pending_rounds.cc:62] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Trying to abort 1 pending ops.
I20250109 20:52:46.090247 1166 pending_rounds.cc:69] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523636952993792 op_type: NO_OP noop_request { }
I20250109 20:52:46.090344 1166 raft_consensus.cc:2883] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:46.090441 1166 raft_consensus.cc:2267] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.088373 1157 log.cc:1198] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4f5a90e65e80494a8b451f2bc6baa2b1
I20250109 20:52:46.097132 1157 ts_tablet_manager.cc:1939] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:46.098245 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4f5a90e65e80494a8b451f2bc6baa2b1 (table default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]) successfully deleted
I20250109 20:52:46.100589 1191 raft_consensus.cc:491] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:46.100752 1191 raft_consensus.cc:513] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.101394 1191 leader_election.cc:290] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:46.102149 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b221e048f6f6463e84e630386c304a05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b221e048f6f6463e84e630386c304a05 already in progress: creating tablet
I20250109 20:52:46.102140 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f2f62aab954531ad6eb60b6f4661b8" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:46.102167 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f2f62aab954531ad6eb60b6f4661b8" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:46.102357 777 raft_consensus.cc:2463] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:46.102463 1157 tablet_replica.cc:331] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:46.102432 1054 raft_consensus.cc:2463] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:46.102607 1157 raft_consensus.cc:2238] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.102764 1157 raft_consensus.cc:2267] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.102900 854 leader_election.cc:304] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:46.103392 1157 ts_tablet_manager.cc:1905] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.103762 1191 raft_consensus.cc:2798] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:46.103868 1191 raft_consensus.cc:491] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:46.103955 1191 raft_consensus.cc:3054] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:46.105695 1158 tablet_bootstrap.cc:492] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:46.105691 1166 ts_tablet_manager.cc:1905] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.105829 1158 ts_tablet_manager.cc:1397] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.002s
I20250109 20:52:46.106609 1158 raft_consensus.cc:357] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.106856 1158 raft_consensus.cc:383] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.106751 1191 raft_consensus.cc:513] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.106917 1158 raft_consensus.cc:738] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.107151 1158 consensus_queue.cc:260] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.107439 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f73279e617714f3e8b0e00b5e8773aeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.107954 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f73279e617714f3e8b0e00b5e8773aeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.110240 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d7539cf9ece4befaae6c521d471ea57 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.110658 1158 ts_tablet_manager.cc:1428] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:46.111253 1166 ts_tablet_manager.cc:1918] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:46.111362 1166 log.cc:1198] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4f5a90e65e80494a8b451f2bc6baa2b1
I20250109 20:52:46.111584 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f2f62aab954531ad6eb60b6f4661b8" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:46.111719 1166 ts_tablet_manager.cc:1939] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.111771 777 raft_consensus.cc:3054] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:46.112617 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4f5a90e65e80494a8b451f2bc6baa2b1 (table default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]) successfully deleted
I20250109 20:52:46.113206 1158 tablet_bootstrap.cc:492] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:46.114308 777 raft_consensus.cc:2463] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:46.115065 1158 tablet_bootstrap.cc:654] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.115046 854 leader_election.cc:304] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2; no voters:
I20250109 20:52:46.115361 1351 raft_consensus.cc:2798] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:46.117532 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f2f62aab954531ad6eb60b6f4661b8" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:46.117726 1054 raft_consensus.cc:3054] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:46.118360 1191 leader_election.cc:290] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:46.120445 1054 raft_consensus.cc:2463] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:46.122179 1296 server_base.cc:1034] running on GCE node
I20250109 20:52:46.127019 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba (DEFAULT_TABLE table=default.Table_e677fcf0892746659b8f6d9f5dce6bc1 [id=d8d809ef804f4f909b825f92a823fb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.127507 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.128759 1166 tablet_replica.cc:331] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:46.128963 1166 raft_consensus.cc:2238] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.129104 1166 raft_consensus.cc:2267] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.129575 1166 ts_tablet_manager.cc:1905] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.123083 1165 tablet_bootstrap.cc:492] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:46.132500 1165 ts_tablet_manager.cc:1397] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.171s user 0.007s sys 0.003s
I20250109 20:52:46.135627 1165 raft_consensus.cc:357] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.135836 1165 raft_consensus.cc:383] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.135900 1165 raft_consensus.cc:738] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.136135 1165 consensus_queue.cc:260] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.136552 1165 ts_tablet_manager.cc:1428] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:52:46.136842 1165 tablet_bootstrap.cc:492] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:46.139194 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9817e2e17d4d4c5ebcbf9ee51fae9654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.139765 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9817e2e17d4d4c5ebcbf9ee51fae9654 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.141690 1157 ts_tablet_manager.cc:1918] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.141795 1157 log.cc:1198] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/175bf6c335a8405c97b296cf4718610b
I20250109 20:52:46.142146 1157 ts_tablet_manager.cc:1939] T 175bf6c335a8405c97b296cf4718610b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:46.140614 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a58d439254f8418091e3f23c8c274240
I20250109 20:52:46.143381 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet a58d439254f8418091e3f23c8c274240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.144129 742 tablet_service.cc:1514] Processing DeleteTablet for tablet a58d439254f8418091e3f23c8c274240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.144785 1165 tablet_bootstrap.cc:654] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.144855 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba (DEFAULT_TABLE table=default.Table_e677fcf0892746659b8f6d9f5dce6bc1 [id=d8d809ef804f4f909b825f92a823fb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.144917 1166 ts_tablet_manager.cc:1918] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.145061 1166 log.cc:1198] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:46.145252 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.145450 1166 ts_tablet_manager.cc:1939] T cc174d6bd9f44eaf9f66e54693597601 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.146413 1158 tablet_bootstrap.cc:492] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:46.146529 1158 ts_tablet_manager.cc:1397] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20250109 20:52:46.147259 1158 raft_consensus.cc:357] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.147433 1158 raft_consensus.cc:383] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.147496 1158 raft_consensus.cc:738] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.147696 1158 consensus_queue.cc:260] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.148898 1158 ts_tablet_manager.cc:1428] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:46.140883 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cba68160be6145ddbbd2019c9b5f0076
I20250109 20:52:46.140966 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 95735385d5ab4fe0b29e33bad14eb95c
I20250109 20:52:46.149478 1158 tablet_bootstrap.cc:492] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:46.151428 1176 tablet_bootstrap.cc:492] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:46.151499 1158 tablet_bootstrap.cc:654] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.151574 1176 ts_tablet_manager.cc:1397] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.220s user 0.019s sys 0.019s
I20250109 20:52:46.152418 1176 raft_consensus.cc:357] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.152621 1176 raft_consensus.cc:383] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.152725 1176 raft_consensus.cc:738] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.153318 1166 tablet_replica.cc:331] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:46.153631 1166 raft_consensus.cc:2238] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.153911 1166 raft_consensus.cc:2267] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.154564 1166 ts_tablet_manager.cc:1905] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.155525 1268 raft_consensus.cc:491] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:46.155661 1268 raft_consensus.cc:513] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.156169 1268 leader_election.cc:290] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:46.157092 713 leader_election.cc:343] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:46.157637 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc174d6bd9f44eaf9f66e54693597601" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:46.158298 712 leader_election.cc:343] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.158412 712 leader_election.cc:304] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:46.158798 1268 raft_consensus.cc:2743] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:46.159029 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 175bf6c335a8405c97b296cf4718610b (table default.Table_5d1e5a64991f49a48a7b4286fc1ffe5f [id=a6f0a5fcb43d4509a188259adfdd0192]) successfully deleted
I20250109 20:52:46.160166 1351 raft_consensus.cc:695] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Running, Role: LEADER
I20250109 20:52:46.154803 1176 consensus_queue.cc:260] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.161610 753 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.161880 1157 tablet_replica.cc:331] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:46.161732 1351 consensus_queue.cc:237] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.162165 1157 raft_consensus.cc:2238] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:46.162319 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 66e4bfa188b545349f358adfd0bc714b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66e4bfa188b545349f358adfd0bc714b already in progress: creating tablet
W20250109 20:52:46.162530 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 91dbccae0af24a629600a92e7eadfee7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91dbccae0af24a629600a92e7eadfee7 already in progress: creating tablet
W20250109 20:52:46.163357 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a58d439254f8418091e3f23c8c274240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a58d439254f8418091e3f23c8c274240 already in progress: creating tablet
I20250109 20:52:46.163973 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet cc174d6bd9f44eaf9f66e54693597601 (table default.Table_c576a7a39c174fe6ac5c351e1b4f80d3 [id=37a12f94e9c54332af6b774b56b13666]) successfully deleted
W20250109 20:52:46.165108 1368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250109 20:52:46.165341 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet fbcadec0c3184feaad4a883e5588f310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcadec0c3184feaad4a883e5588f310 already in progress: creating tablet
I20250109 20:52:46.165808 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7609a405eec042e4982438d13650f8a4" } new_table_name: "default.Table_12b05f4d86f446c884e83c9f983d5ca2" modify_external_catalogs: true
I20250109 20:52:46.166186 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170]). Expected version 2 got 0
I20250109 20:52:46.167205 1157 raft_consensus.cc:2267] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.164206 742 tablet_service.cc:1514] Processing DeleteTablet for tablet cba68160be6145ddbbd2019c9b5f0076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.164412 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.164588 753 tablet_service.cc:1467] Processing CreateTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e (DEFAULT_TABLE table=default.Table_c762911889354a9497c8d731f5e6c578 [id=6aa6003be0684dc88b8ed001eed73727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.167799 1296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:52:46.167930 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f502d666aa043e2a0b67a4e21bcf03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.169044 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7d7539cf9ece4befaae6c521d471ea57
I20250109 20:52:46.169253 750 tablet_service.cc:1467] Processing CreateTablet for tablet 59e1e8754625463ea42a92ef65a89214 (DEFAULT_TABLE table=default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.169587 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e1e8754625463ea42a92ef65a89214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.171371 451 catalog_manager.cc:6596] ReplaceTablet: tablet 7d7539cf9ece4befaae6c521d471ea57 deleted and replaced by tablet 0b0450d88e3e4ccaae4e5157ce5c80c1
I20250109 20:52:46.171864 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b0450d88e3e4ccaae4e5157ce5c80c1) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.173759 742 tablet_service.cc:1467] Processing CreateTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 (DEFAULT_TABLE table=default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.174126 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ed00f7954143c9ba9bf65d8a8a68d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.174281 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba (DEFAULT_TABLE table=default.Table_e677fcf0892746659b8f6d9f5dce6bc1 [id=d8d809ef804f4f909b825f92a823fb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.174674 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.177556 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.177739 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e (DEFAULT_TABLE table=default.Table_c762911889354a9497c8d731f5e6c578 [id=6aa6003be0684dc88b8ed001eed73727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.178624 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f502d666aa043e2a0b67a4e21bcf03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.178663 1157 ts_tablet_manager.cc:1905] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:46.180191 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 540531463c184b9abf226d7487939e60 consensus state has no leader
I20250109 20:52:46.180298 1165 tablet_bootstrap.cc:492] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:46.180408 1165 ts_tablet_manager.cc:1397] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.002s
I20250109 20:52:46.181128 1165 raft_consensus.cc:357] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.181342 1165 raft_consensus.cc:383] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.181416 1165 raft_consensus.cc:738] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.181605 1165 consensus_queue.cc:260] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.182092 1165 ts_tablet_manager.cc:1428] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:46.182296 741 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8d67b624984a0bb327d045c55476da (DEFAULT_TABLE table=default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.183182 754 tablet_service.cc:1467] Processing CreateTablet for tablet cda300a7e58a44339c3bdb2e62a9f384 (DEFAULT_TABLE table=default.Table_6e345f747be74c0c9cfb28cf6ff90e93 [id=1284f08f96184576bde56fa61834e33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.183611 893 tablet_service.cc:1514] Processing DeleteTablet for tablet a58d439254f8418091e3f23c8c274240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.184888 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9498c3dc02794da496271a51706a9c7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.185333 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9498c3dc02794da496271a51706a9c7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.186465 1296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:52:46.182443 751 tablet_service.cc:1467] Processing CreateTablet for tablet aa7d7695ff5946abb6ee576431d2e87f (DEFAULT_TABLE table=default.Table_d311b482665944949ba9d26a91d7fd39 [id=d1eb369ad1da4221b0be6419fe55ad33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.187048 893 tablet_service.cc:1467] Processing CreateTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e (DEFAULT_TABLE table=default.Table_c762911889354a9497c8d731f5e6c578 [id=6aa6003be0684dc88b8ed001eed73727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.187335 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d8d67b624984a0bb327d045c55476da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.187604 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f502d666aa043e2a0b67a4e21bcf03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.188849 750 tablet_service.cc:1467] Processing CreateTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 (DEFAULT_TABLE table=default.Table_5e3e729549234c39a601e29da97094d4 [id=5811deb4d7214be490d07cf674920448]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.188943 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet cba68160be6145ddbbd2019c9b5f0076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.189285 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa7d7695ff5946abb6ee576431d2e87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.189950 1296 hybrid_clock.cc:648] HybridClock initialized: now 1736455966189875 us; error 36 us; skew 500 ppm
I20250109 20:52:46.190407 1296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250109 20:52:46.192287 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 34f215eee55d4e6a97a384df6ff9d596 (table default.Table_8f9fc791c58d4d3fbd9ad92c0a88003f [id=fc92038a07574bcf9e5c970ba84eac8c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 34f215eee55d4e6a97a384df6ff9d596 consensus state has no leader
I20250109 20:52:46.194211 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cda300a7e58a44339c3bdb2e62a9f384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.194099 451 catalog_manager.cc:5526] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d473955919435cb679edd5f8afe2c2 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "44d473955919435cb679edd5f8afe2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:46.197556 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.198256 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e677fcf0892746659b8f6d9f5dce6bc1" } modify_external_catalogs: true
I20250109 20:52:46.198437 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e677fcf0892746659b8f6d9f5dce6bc1" } modify_external_catalogs: true
I20250109 20:52:46.198859 893 tablet_service.cc:1514] Processing DeleteTablet for tablet cba68160be6145ddbbd2019c9b5f0076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.199519 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06c3aa20539740cba4e5d6e852ad2909. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.200325 892 tablet_service.cc:1467] Processing CreateTablet for tablet 59e1e8754625463ea42a92ef65a89214 (DEFAULT_TABLE table=default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.200698 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e1e8754625463ea42a92ef65a89214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.200767 1165 tablet_bootstrap.cc:492] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:46.201215 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_592b3f74ec6c48fdb0ac9daff1341dd7" } new_table_name: "default.Table_68ffdf1aeefe420ea24aa1102012f77c" modify_external_catalogs: true
I20250109 20:52:46.202276 1165 tablet_bootstrap.cc:654] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.204129 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c86baf1a40a34eb4beeff2b71b73afde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.204643 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c86baf1a40a34eb4beeff2b71b73afde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.205627 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b0450d88e3e4ccaae4e5157ce5c80c1) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.205979 890 tablet_service.cc:1467] Processing CreateTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 (DEFAULT_TABLE table=default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.206043 885 tablet_service.cc:1467] Processing CreateTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 (DEFAULT_TABLE table=default.Table_5e3e729549234c39a601e29da97094d4 [id=5811deb4d7214be490d07cf674920448]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.206305 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ed00f7954143c9ba9bf65d8a8a68d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.206266 899 tablet_service.cc:1467] Processing CreateTablet for tablet aa7d7695ff5946abb6ee576431d2e87f (DEFAULT_TABLE table=default.Table_d311b482665944949ba9d26a91d7fd39 [id=d1eb369ad1da4221b0be6419fe55ad33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.206517 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa7d7695ff5946abb6ee576431d2e87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.206629 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c762911889354a9497c8d731f5e6c578" } new_table_name: "default.Table_b086cd04a8e249a09d729be46eda67b1" modify_external_catalogs: true
I20250109 20:52:46.206709 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8d67b624984a0bb327d045c55476da (DEFAULT_TABLE table=default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.206926 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06c3aa20539740cba4e5d6e852ad2909. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.207146 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d8d67b624984a0bb327d045c55476da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.213744 1176 ts_tablet_manager.cc:1428] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250109 20:52:46.214133 451 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170]). Expected version 2 got 0
I20250109 20:52:46.214324 893 tablet_service.cc:1467] Processing CreateTablet for tablet cda300a7e58a44339c3bdb2e62a9f384 (DEFAULT_TABLE table=default.Table_6e345f747be74c0c9cfb28cf6ff90e93 [id=1284f08f96184576bde56fa61834e33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.214697 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cda300a7e58a44339c3bdb2e62a9f384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.216501 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8a5a6dbb823e4bd3afeb157b3c5771ba
I20250109 20:52:46.219576 1027 tablet_service.cc:1467] Processing CreateTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 (DEFAULT_TABLE table=default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.219938 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ed00f7954143c9ba9bf65d8a8a68d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.222724 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 59e1e8754625463ea42a92ef65a89214 (DEFAULT_TABLE table=default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.223027 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e1e8754625463ea42a92ef65a89214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.223351 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 (DEFAULT_TABLE table=default.Table_5e3e729549234c39a601e29da97094d4 [id=5811deb4d7214be490d07cf674920448]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.223906 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b0450d88e3e4ccaae4e5157ce5c80c1) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.224097 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.224550 1024 tablet_service.cc:1467] Processing CreateTablet for tablet aa7d7695ff5946abb6ee576431d2e87f (DEFAULT_TABLE table=default.Table_d311b482665944949ba9d26a91d7fd39 [id=d1eb369ad1da4221b0be6419fe55ad33]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:46.224817 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1c5a6b9681d4ef6a36e308ab2916423 has no consensus state
I20250109 20:52:46.224824 1028 tablet_service.cc:1467] Processing CreateTablet for tablet cda300a7e58a44339c3bdb2e62a9f384 (DEFAULT_TABLE table=default.Table_6e345f747be74c0c9cfb28cf6ff90e93 [id=1284f08f96184576bde56fa61834e33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.225267 1187 raft_consensus.cc:491] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:46.225397 1187 raft_consensus.cc:513] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.226507 1187 leader_election.cc:290] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:46.227550 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eb4dfcf96c4a05a55082e90ae04467" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:46.228883 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06c3aa20539740cba4e5d6e852ad2909. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.231581 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a58d439254f8418091e3f23c8c274240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a58d439254f8418091e3f23c8c274240 already in progress: creating tablet
W20250109 20:52:46.231868 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 95735385d5ab4fe0b29e33bad14eb95c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95735385d5ab4fe0b29e33bad14eb95c already in progress: creating tablet
W20250109 20:52:46.232209 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cba68160be6145ddbbd2019c9b5f0076 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cba68160be6145ddbbd2019c9b5f0076 already in progress: creating tablet
W20250109 20:52:46.232435 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7d7539cf9ece4befaae6c521d471ea57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d7539cf9ece4befaae6c521d471ea57 already in progress: creating tablet
I20250109 20:52:46.236574 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa7d7695ff5946abb6ee576431d2e87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.236594 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8d67b624984a0bb327d045c55476da (DEFAULT_TABLE table=default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.239719 1165 tablet_bootstrap.cc:492] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:46.239920 1165 ts_tablet_manager.cc:1397] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.000s
W20250109 20:52:46.240653 989 leader_election.cc:343] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.240860 1026 tablet_service.cc:1467] Processing CreateTablet for tablet cd596614521a4a92a71b781b162f70b6 (DEFAULT_TABLE table=default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.241297 891 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1891cbef8f4b0cae0c15d478db4860 (DEFAULT_TABLE table=default.Table_db3d6cbe91b54958b543da192a338eb5 [id=0a0599668d77405ba0e72eda2aa5219e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.241576 890 tablet_service.cc:1467] Processing CreateTablet for tablet cd596614521a4a92a71b781b162f70b6 (DEFAULT_TABLE table=default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.241720 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b1891cbef8f4b0cae0c15d478db4860. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.242079 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd596614521a4a92a71b781b162f70b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.242287 455 master_service.cc:946] ReplaceTablet: received request to replace tablet ffd6ce78438841daa663b6344f062019 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.242412 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd596614521a4a92a71b781b162f70b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.244326 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d8d67b624984a0bb327d045c55476da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.245519 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1891cbef8f4b0cae0c15d478db4860 (DEFAULT_TABLE table=default.Table_db3d6cbe91b54958b543da192a338eb5 [id=0a0599668d77405ba0e72eda2aa5219e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.245641 885 tablet_service.cc:1467] Processing CreateTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 (DEFAULT_TABLE table=default.Table_bf29c4a64f7b4d49af314ad6211982b1 [id=7d02be4e89a7426ea835a9041c394fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.245937 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f67c4294f8499496db4e82db8c3bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.246434 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b1891cbef8f4b0cae0c15d478db4860. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.246984 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f502d666aa043e2a0b67a4e21bcf03e consensus state has no leader
I20250109 20:52:46.247319 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce64e0f3a004c9387bb592cf19236e7 (DEFAULT_TABLE table=default.Table_6636fd244b2448a4a8de2eda78a535a4 [id=f91ce8749a344785a06059e5e4fa97b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.247963 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 (DEFAULT_TABLE table=default.Table_bf29c4a64f7b4d49af314ad6211982b1 [id=7d02be4e89a7426ea835a9041c394fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.249404 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cda300a7e58a44339c3bdb2e62a9f384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.250114 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce64e0f3a004c9387bb592cf19236e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.250356 745 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1891cbef8f4b0cae0c15d478db4860 (DEFAULT_TABLE table=default.Table_db3d6cbe91b54958b543da192a338eb5 [id=0a0599668d77405ba0e72eda2aa5219e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.250697 1176 tablet_bootstrap.cc:492] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:46.250721 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b1891cbef8f4b0cae0c15d478db4860. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.252621 742 tablet_service.cc:1467] Processing CreateTablet for tablet cd596614521a4a92a71b781b162f70b6 (DEFAULT_TABLE table=default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.252993 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd596614521a4a92a71b781b162f70b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.253788 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f5a90e65e80494a8b451f2bc6baa2b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b5ba73bbb08a4db0adfc9e72a84a22e8) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.254915 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f67c4294f8499496db4e82db8c3bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.255944 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.256287 1158 tablet_bootstrap.cc:492] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:46.256302 892 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce64e0f3a004c9387bb592cf19236e7 (DEFAULT_TABLE table=default.Table_6636fd244b2448a4a8de2eda78a535a4 [id=f91ce8749a344785a06059e5e4fa97b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.256743 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce64e0f3a004c9387bb592cf19236e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.257382 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eb4dfcf96c4a05a55082e90ae04467" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:46.257707 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.259832 756 tablet_service.cc:1467] Processing CreateTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 (DEFAULT_TABLE table=default.Table_bf29c4a64f7b4d49af314ad6211982b1 [id=7d02be4e89a7426ea835a9041c394fd4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:46.259172 990 leader_election.cc:343] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.260660 990 leader_election.cc:304] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:46.261071 1187 raft_consensus.cc:2743] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:46.261085 898 tablet_service.cc:1467] Processing CreateTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 (DEFAULT_TABLE table=default.Table_68ffdf1aeefe420ea24aa1102012f77c [id=4d826d307bff4ab2b43a9d8b718055d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.261387 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c5a6b9681d4ef6a36e308ab2916423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.262085 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 (DEFAULT_TABLE table=default.Table_f7702e93eb894ea5bbf0af322f94c29c [id=0193ea72702947b590c27b63fd70f93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.263942 890 tablet_service.cc:1467] Processing CreateTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 (DEFAULT_TABLE table=default.Table_9817e2e17d4d4c5ebcbf9ee51fae9654 [id=5c7e66cff6b14f9a81029cea36224460]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.264056 1165 raft_consensus.cc:357] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.264328 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219a4b508050490cb3f1ee418e0edbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.264394 1165 raft_consensus.cc:383] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.264492 1165 raft_consensus.cc:738] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.264803 1165 consensus_queue.cc:260] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.266041 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d311b482665944949ba9d26a91d7fd39" } modify_external_catalogs: true
I20250109 20:52:46.266294 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d311b482665944949ba9d26a91d7fd39" } modify_external_catalogs: true
I20250109 20:52:46.267582 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_70499a7567fa4d6498705309cde3b257"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.269505 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f7166988e8c743c9896cbdbd048fa145" } new_table_name: "default.Table_bb480bfd701049cc8b814e0fe8b4c9ce" modify_external_catalogs: true
I20250109 20:52:46.270566 1165 ts_tablet_manager.cc:1428] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20250109 20:52:46.271292 1165 tablet_bootstrap.cc:492] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:46.272697 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0a104f68c5934305bc28de2aa457e751"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.272887 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f67c4294f8499496db4e82db8c3bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.274358 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0538f4d3608a454b9dd21a0fa42068a0 (DEFAULT_TABLE table=default.Table_9498c3dc02794da496271a51706a9c7a [id=3fde04a3af42412db45f2c91950fb8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.274565 1020 tablet_service.cc:1467] Processing CreateTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 (DEFAULT_TABLE table=default.Table_68ffdf1aeefe420ea24aa1102012f77c [id=4d826d307bff4ab2b43a9d8b718055d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.274716 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e82535780c47ca8b9cd94b013b4b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.275674 748 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca83a49523b45be9ac7556b40746a83 (DEFAULT_TABLE table=default.Table_79f5ff4b60444c1fb9e227b39c89e4f4 [id=523cbdd3d4344d24afca6003e115d5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.276968 888 tablet_service.cc:1467] Processing CreateTablet for tablet 35a39fac67894a208dd7ec3153b05e4a (DEFAULT_TABLE table=default.Table_f73279e617714f3e8b0e00b5e8773aeb [id=fcb76535fbbe44e29ef627a3ae562ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.273597 740 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce64e0f3a004c9387bb592cf19236e7 (DEFAULT_TABLE table=default.Table_6636fd244b2448a4a8de2eda78a535a4 [id=f91ce8749a344785a06059e5e4fa97b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:46.278119 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70499a7567fa4d6498705309cde3b257 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:46.279759 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6716e7428ce240fca7ca5b02f564a5df consensus state has no leader
W20250109 20:52:46.281148 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a104f68c5934305bc28de2aa457e751 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.283367 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca83a49523b45be9ac7556b40746a83 (DEFAULT_TABLE table=default.Table_79f5ff4b60444c1fb9e227b39c89e4f4 [id=523cbdd3d4344d24afca6003e115d5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.283910 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 (DEFAULT_TABLE table=default.Table_9817e2e17d4d4c5ebcbf9ee51fae9654 [id=5c7e66cff6b14f9a81029cea36224460]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.284041 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a35c15eced344d6fb5e7a4322a01a160" } new_table_name: "default.Table_dcd777ce1ff24c3db7dafe776cf3b871" modify_external_catalogs: true
I20250109 20:52:46.285390 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0bef1709ef9c47738d52c3d41d4f6819"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.285833 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bef1709ef9c47738d52c3d41d4f6819 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.286830 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e9b31b27949c2948cb64b6124ee72 (DEFAULT_TABLE table=default.Table_8b85547ea1c642a693a94f23f77df03f [id=d320c4ac79d046479cf0718148b62b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.290271 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0538f4d3608a454b9dd21a0fa42068a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.290606 1296 webserver.cc:458] Webserver started at http://127.0.104.60:46069/ using document root <none> and password file <none>
I20250109 20:52:46.290930 1158 ts_tablet_manager.cc:1397] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.031s
I20250109 20:52:46.291234 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9c94d3852ae945d0b0dd5798dac99fba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.291697 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c94d3852ae945d0b0dd5798dac99fba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.291916 1296 fs_manager.cc:362] Metadata directory not provided
I20250109 20:52:46.292068 1296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:52:46.288889 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 35a39fac67894a208dd7ec3153b05e4a (DEFAULT_TABLE table=default.Table_f73279e617714f3e8b0e00b5e8773aeb [id=fcb76535fbbe44e29ef627a3ae562ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.294289 885 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e9b31b27949c2948cb64b6124ee72 (DEFAULT_TABLE table=default.Table_8b85547ea1c642a693a94f23f77df03f [id=d320c4ac79d046479cf0718148b62b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.295416 1160 tablet_replica.cc:331] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:46.296031 749 tablet_service.cc:1467] Processing CreateTablet for tablet 35a39fac67894a208dd7ec3153b05e4a (DEFAULT_TABLE table=default.Table_f73279e617714f3e8b0e00b5e8773aeb [id=fcb76535fbbe44e29ef627a3ae562ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.297027 1176 tablet_bootstrap.cc:654] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.298074 1158 raft_consensus.cc:357] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.298738 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:46.299024 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d1630e4438e4337ba026b41f630a480 consensus state has no leader
I20250109 20:52:46.300311 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 (DEFAULT_TABLE table=default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.301815 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_03765feae81a4760b3e7064beec92515"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.302448 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a39fac67894a208dd7ec3153b05e4a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.302449 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03765feae81a4760b3e7064beec92515 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.305342 752 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 (DEFAULT_TABLE table=default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.306285 746 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 (DEFAULT_TABLE table=default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.307533 744 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e9b31b27949c2948cb64b6124ee72 (DEFAULT_TABLE table=default.Table_8b85547ea1c642a693a94f23f77df03f [id=d320c4ac79d046479cf0718148b62b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.307567 755 tablet_service.cc:1467] Processing CreateTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 (DEFAULT_TABLE table=default.Table_68ffdf1aeefe420ea24aa1102012f77c [id=4d826d307bff4ab2b43a9d8b718055d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.307912 747 tablet_service.cc:1467] Processing CreateTablet for tablet 0538f4d3608a454b9dd21a0fa42068a0 (DEFAULT_TABLE table=default.Table_9498c3dc02794da496271a51706a9c7a [id=3fde04a3af42412db45f2c91950fb8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.308243 739 tablet_service.cc:1467] Processing CreateTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 (DEFAULT_TABLE table=default.Table_9817e2e17d4d4c5ebcbf9ee51fae9654 [id=5c7e66cff6b14f9a81029cea36224460]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.309242 757 tablet_service.cc:1467] Processing CreateTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 (DEFAULT_TABLE table=default.Table_f7702e93eb894ea5bbf0af322f94c29c [id=0193ea72702947b590c27b63fd70f93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.310251 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a39fac67894a208dd7ec3153b05e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.311358 1158 raft_consensus.cc:383] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.311450 1158 raft_consensus.cc:738] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.311695 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c5a6b9681d4ef6a36e308ab2916423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.312321 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e82535780c47ca8b9cd94b013b4b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.312157 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a137a6b3e0614575b0a4f169e928ce95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.312798 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a137a6b3e0614575b0a4f169e928ce95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.313715 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.314517 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0538f4d3608a454b9dd21a0fa42068a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.315680 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce64e0f3a004c9387bb592cf19236e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.316224 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.317445 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 (DEFAULT_TABLE table=default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.318482 1158 consensus_queue.cc:260] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.319092 1158 ts_tablet_manager.cc:1428] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20250109 20:52:46.319379 1158 tablet_bootstrap.cc:492] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:46.322166 1157 ts_tablet_manager.cc:1918] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.322988 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c5a6b9681d4ef6a36e308ab2916423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.324033 1157 log.cc:1198] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:46.324421 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5e9b31b27949c2948cb64b6124ee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.325347 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c9f7274a7542e89a19bd4afcf89205. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.327454 1160 raft_consensus.cc:2238] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.329157 899 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca83a49523b45be9ac7556b40746a83 (DEFAULT_TABLE table=default.Table_79f5ff4b60444c1fb9e227b39c89e4f4 [id=523cbdd3d4344d24afca6003e115d5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.331291 887 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 (DEFAULT_TABLE table=default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.331880 897 tablet_service.cc:1467] Processing CreateTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 (DEFAULT_TABLE table=default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.334203 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 97205d09de264adb9d19cb32546b0942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.334513 1166 ts_tablet_manager.cc:1918] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:46.334661 1166 log.cc:1198] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/38760b86fdfb4c8c8bcfd90874cbfd0d
I20250109 20:52:46.336632 1157 ts_tablet_manager.cc:1939] T cc174d6bd9f44eaf9f66e54693597601 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:46.338425 893 tablet_service.cc:1467] Processing CreateTablet for tablet 0538f4d3608a454b9dd21a0fa42068a0 (DEFAULT_TABLE table=default.Table_9498c3dc02794da496271a51706a9c7a [id=3fde04a3af42412db45f2c91950fb8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.338976 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca83a49523b45be9ac7556b40746a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.339030 1160 raft_consensus.cc:2267] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.339270 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet cc174d6bd9f44eaf9f66e54693597601 (table default.Table_c576a7a39c174fe6ac5c351e1b4f80d3 [id=37a12f94e9c54332af6b774b56b13666]) successfully deleted
I20250109 20:52:46.339677 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219a4b508050490cb3f1ee418e0edbf8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.339905 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cba68160be6145ddbbd2019c9b5f0076 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cba68160be6145ddbbd2019c9b5f0076 already in progress: creating tablet
W20250109 20:52:46.340410 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 95735385d5ab4fe0b29e33bad14eb95c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95735385d5ab4fe0b29e33bad14eb95c already in progress: creating tablet
W20250109 20:52:46.341279 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7d7539cf9ece4befaae6c521d471ea57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d7539cf9ece4befaae6c521d471ea57 already in progress: creating tablet
W20250109 20:52:46.341768 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a5a6dbb823e4bd3afeb157b3c5771ba already in progress: creating tablet
I20250109 20:52:46.342031 1157 tablet_replica.cc:331] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:46.342201 1157 raft_consensus.cc:2238] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.342381 1157 raft_consensus.cc:2267] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.342792 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c9f7274a7542e89a19bd4afcf89205. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.342965 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5e9b31b27949c2948cb64b6124ee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.345504 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219a4b508050490cb3f1ee418e0edbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.346727 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca83a49523b45be9ac7556b40746a83. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.346753 1191 raft_consensus.cc:668] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20250109 20:52:46.347327 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 175bf6c335a8405c97b296cf4718610b with delete_type TABLET_DATA_DELETED (replaced by tablet 11d303cd42a2428fa17f29b48c4f64f5) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.347702 1166 ts_tablet_manager.cc:1939] T 38760b86fdfb4c8c8bcfd90874cbfd0d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.348477 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet aa7d7695ff5946abb6ee576431d2e87f
I20250109 20:52:46.349958 753 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7d7695ff5946abb6ee576431d2e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.354292 1158 tablet_bootstrap.cc:654] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.354770 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d70feebb4775482eb8b3b251c50ad92c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.355351 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d70feebb4775482eb8b3b251c50ad92c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.356859 1160 ts_tablet_manager.cc:1905] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.357303 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_65cb0c2ed23b450ca192c6f61becbc51" } modify_external_catalogs: true
I20250109 20:52:46.357532 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_65cb0c2ed23b450ca192c6f61becbc51" } modify_external_catalogs: true
I20250109 20:52:46.362529 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 38760b86fdfb4c8c8bcfd90874cbfd0d (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]) successfully deleted
W20250109 20:52:46.362721 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 54d11b9148dc42bc91348e837af73972 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54d11b9148dc42bc91348e837af73972 already in progress: creating tablet
W20250109 20:52:46.363057 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ea5d462230734767bb010f520c86c5ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea5d462230734767bb010f520c86c5ab already in progress: creating tablet
I20250109 20:52:46.364228 1166 tablet_replica.cc:331] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:46.364854 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7d981c7f13c3409f994bfdf0275e0116 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d981c7f13c3409f994bfdf0275e0116 already in progress: creating tablet
W20250109 20:52:46.365096 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f1589b79a56148b6ab2946943de39fb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1589b79a56148b6ab2946943de39fb2 already in progress: creating tablet
W20250109 20:52:46.365269 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 11d303cd42a2428fa17f29b48c4f64f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11d303cd42a2428fa17f29b48c4f64f5 already in progress: creating tablet
I20250109 20:52:46.366014 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a39fac67894a208dd7ec3153b05e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.366251 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ff7c9fe449d540f19d0ce42d44ba0b73" } modify_external_catalogs: true
I20250109 20:52:46.366322 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4792b8689a12402f9946e87ad47dcbbb" } modify_external_catalogs: true
I20250109 20:52:46.366434 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ff7c9fe449d540f19d0ce42d44ba0b73" } modify_external_catalogs: true
I20250109 20:52:46.366448 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4792b8689a12402f9946e87ad47dcbbb" } modify_external_catalogs: true
I20250109 20:52:46.368887 891 tablet_service.cc:1467] Processing CreateTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 (DEFAULT_TABLE table=default.Table_f7702e93eb894ea5bbf0af322f94c29c [id=0193ea72702947b590c27b63fd70f93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.369920 1166 raft_consensus.cc:2238] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.372404 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7d7695ff5946abb6ee576431d2e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.372772 1166 raft_consensus.cc:2267] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.378983 1187 raft_consensus.cc:491] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:46.379141 1187 raft_consensus.cc:513] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.379474 1166 ts_tablet_manager.cc:1905] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.382269 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044c212aada4df382d147c9a15ea4a7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:46.383008 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8f9fc791c58d4d3fbd9ad92c0a88003f" } new_table_name: "default.Table_31d5f3a5564a4159a75be95e73dd5bf5" modify_external_catalogs: true
I20250109 20:52:46.383510 1187 leader_election.cc:290] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:52:46.384613 990 leader_election.cc:343] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.391512 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 41de7acfa7f941caa94bdbb888cf841e
I20250109 20:52:46.393821 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de7acfa7f941caa94bdbb888cf841e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.394129 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de7acfa7f941caa94bdbb888cf841e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.397325 1187 raft_consensus.cc:491] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:46.397488 1187 raft_consensus.cc:513] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.398120 1187 leader_election.cc:290] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:46.398214 1157 ts_tablet_manager.cc:1905] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.402225 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2d96a569e6be4e01a26918d115cfbcf9
I20250109 20:52:46.405880 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de7acfa7f941caa94bdbb888cf841e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.406113 889 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7d7695ff5946abb6ee576431d2e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.406141 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1ee95fba580446b8926cbc779fca720f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.406257 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1044c212aada4df382d147c9a15ea4a7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:46.406594 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5e9b31b27949c2948cb64b6124ee72. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.406802 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ee95fba580446b8926cbc779fca720f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.407392 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0538f4d3608a454b9dd21a0fa42068a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.408021 989 leader_election.cc:343] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.408135 989 leader_election.cc:304] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:46.408510 1187 raft_consensus.cc:2743] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:46.410229 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.411571 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f865acd2c9aa47ec8ebc3bf90d81c21e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:46.411828 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.412146 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6acca65ecee94f8cbe3f2b89e744aa6c
W20250109 20:52:46.412540 989 leader_election.cc:343] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.412810 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c9f7274a7542e89a19bd4afcf89205. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.414202 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.416275 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca83a49523b45be9ac7556b40746a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.419723 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:46.419855 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34f215eee55d4e6a97a384df6ff9d596 consensus state has no leader
I20250109 20:52:46.420487 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6de1d3cd236842a3a239ec4c275d4986"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.420981 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6de1d3cd236842a3a239ec4c275d4986 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.422617 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 (DEFAULT_TABLE table=default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.422855 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d59d31f7bc514ff38a99d27cbe8170f9" } modify_external_catalogs: true
I20250109 20:52:46.423007 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8890a6a93e3d48dc96955f1b51e78603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.423015 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d59d31f7bc514ff38a99d27cbe8170f9" } modify_external_catalogs: true
I20250109 20:52:46.427466 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_14ad6cd05135486e87607f62e9dd62c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.427894 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14ad6cd05135486e87607f62e9dd62c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.428154 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e82535780c47ca8b9cd94b013b4b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.428836 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f865acd2c9aa47ec8ebc3bf90d81c21e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:46.430115 990 leader_election.cc:343] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.430292 990 leader_election.cc:304] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:46.430524 892 tablet_service.cc:1467] Processing CreateTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 (DEFAULT_TABLE table=default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.430784 1187 raft_consensus.cc:2743] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:46.432873 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_96663f02560b43c1bef9c2cad86ae6c8" } new_table_name: "default.Table_579b6c1d61dd4e80af5ea30b4e06dc35" modify_external_catalogs: true
I20250109 20:52:46.436525 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.440675 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 49c6f1833ff1453f90a6ec6435248a02
I20250109 20:52:46.440635 1296 fs_manager.cc:714] Time spent opening directory manager: real 0.146s user 0.014s sys 0.046s
W20250109 20:52:46.441471 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c40e5019321c497b814a281e0ab1a7f7 consensus state has no leader
I20250109 20:52:46.441462 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.443394 1176 tablet_bootstrap.cc:492] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:46.443660 1176 ts_tablet_manager.cc:1397] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.229s user 0.003s sys 0.011s
I20250109 20:52:46.444456 1176 raft_consensus.cc:357] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.444788 1166 ts_tablet_manager.cc:1918] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.444658 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.444991 1176 raft_consensus.cc:383] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.445144 1176 raft_consensus.cc:738] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.445434 1176 consensus_queue.cc:260] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.445919 1176 ts_tablet_manager.cc:1428] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:46.445134 1166 log.cc:1198] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/825efab5ff7b492ca3c7e3fe11ece098
I20250109 20:52:46.446333 1166 ts_tablet_manager.cc:1939] T 825efab5ff7b492ca3c7e3fe11ece098 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.446782 1176 tablet_bootstrap.cc:492] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:46.448397 1176 tablet_bootstrap.cc:654] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.448829 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_31d5f3a5564a4159a75be95e73dd5bf5" } new_table_name: "default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4" modify_external_catalogs: true
I20250109 20:52:46.451627 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.453428 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.454218 1166 tablet_replica.cc:331] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:46.454322 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.454450 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 825efab5ff7b492ca3c7e3fe11ece098 (table default.Table_7a97ba56daf04341bc72ea1ffe462b8c [id=9c14de13a61b45f78bacd804507a2b14]) successfully deleted
I20250109 20:52:46.456177 1166 raft_consensus.cc:2238] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.456358 1166 raft_consensus.cc:2267] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.456869 1166 ts_tablet_manager.cc:1905] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:46.462874 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34f215eee55d4e6a97a384df6ff9d596 consensus state has no leader
I20250109 20:52:46.466511 1176 tablet_bootstrap.cc:492] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:46.466873 1176 ts_tablet_manager.cc:1397] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250109 20:52:46.468992 1176 raft_consensus.cc:357] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.470342 1176 raft_consensus.cc:383] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.470436 1176 raft_consensus.cc:738] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.470688 1176 consensus_queue.cc:260] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.472123 1176 ts_tablet_manager.cc:1428] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250109 20:52:46.472383 1176 tablet_bootstrap.cc:492] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:46.473929 1176 tablet_bootstrap.cc:654] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.479537 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acadf60d566407aabc1a2395d1d235f with delete_type TABLET_DATA_DELETED (replaced by tablet f672cbd6dffc49dbb69992ae7db979ea) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.484211 1166 ts_tablet_manager.cc:1918] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.486352 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_12b05f4d86f446c884e83c9f983d5ca2" } new_table_name: "default.Table_642aeae240da4ea3a0fca064e8489134" modify_external_catalogs: true
I20250109 20:52:46.488248 1027 tablet_service.cc:1467] Processing CreateTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc (DEFAULT_TABLE table=default.Table_a137a6b3e0614575b0a4f169e928ce95 [id=7adda20a8381463a9f6decd4c7cd045d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.488481 889 tablet_service.cc:1467] Processing CreateTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc (DEFAULT_TABLE table=default.Table_a137a6b3e0614575b0a4f169e928ce95 [id=7adda20a8381463a9f6decd4c7cd045d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.488641 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 (DEFAULT_TABLE table=default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.488844 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd3a1b1d3e44feda6dbaa8556e377fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.489146 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e9b6101e3447fb3c24b3917aa4872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.494696 450 master_service.cc:946] ReplaceTablet: received request to replace tablet ed5ea5d588834f1b8b2b057ec324aadc from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.494622 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 9558dfe563d94482bf496e19b8a80edc (DEFAULT_TABLE table=default.Table_9c94d3852ae945d0b0dd5798dac99fba [id=095b22eda1d34de4b3394189471c53c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.494810 1165 tablet_bootstrap.cc:654] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.494988 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9558dfe563d94482bf496e19b8a80edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.497550 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 (DEFAULT_TABLE table=default.Table_70499a7567fa4d6498705309cde3b257 [id=3ae741f5fe4343188b6f0d10b2a09b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.497884 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4f047e8fa34ca79e78915c61a8ba12. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.500792 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 540531463c184b9abf226d7487939e60 consensus state has no leader
I20250109 20:52:46.506973 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_53b6d02e81214c5ab40407b9d3fa6584" } new_table_name: "default.Table_53a8873bce9e4a749786b79099e7b5b1" modify_external_catalogs: true
I20250109 20:52:46.510486 899 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 (DEFAULT_TABLE table=default.Table_70499a7567fa4d6498705309cde3b257 [id=3ae741f5fe4343188b6f0d10b2a09b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.510649 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4f4640c9222649fc957636279628a391"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.510869 887 tablet_service.cc:1467] Processing CreateTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 (DEFAULT_TABLE table=default.Table_0bef1709ef9c47738d52c3d41d4f6819 [id=65098b4e94ce440494b2d5682132ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:46.511135 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f4640c9222649fc957636279628a391 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.512212 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7d999163ab094d83933d7da655cb380a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.512637 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d999163ab094d83933d7da655cb380a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.512836 890 tablet_service.cc:1467] Processing CreateTablet for tablet 57e56f30e6f3450a8426cd40db89e009 (DEFAULT_TABLE table=default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.512971 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ed5ea5d588834f1b8b2b057ec324aadc
I20250109 20:52:46.513486 450 catalog_manager.cc:6596] ReplaceTablet: tablet ed5ea5d588834f1b8b2b057ec324aadc deleted and replaced by tablet cd23012af5574f37944f9e26c07d05d2
I20250109 20:52:46.516371 898 tablet_service.cc:1467] Processing CreateTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 (DEFAULT_TABLE table=default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.516675 895 tablet_service.cc:1467] Processing CreateTablet for tablet 9558dfe563d94482bf496e19b8a80edc (DEFAULT_TABLE table=default.Table_9c94d3852ae945d0b0dd5798dac99fba [id=095b22eda1d34de4b3394189471c53c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.516973 896 tablet_service.cc:1467] Processing CreateTablet for tablet 93d409409813420689134773c627833b (DEFAULT_TABLE table=default.Table_1ee95fba580446b8926cbc779fca720f [id=a4a7c86d0df2412a83f12157c967aae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.517721 894 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed134863bf642e79c26e5bf75b248b0 (DEFAULT_TABLE table=default.Table_0a104f68c5934305bc28de2aa457e751 [id=46c011650d964e26b40be8a854e1f8c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.518049 886 tablet_service.cc:1467] Processing CreateTablet for tablet daac99da5d10475c9de263bef27baeb7 (DEFAULT_TABLE table=default.Table_d70feebb4775482eb8b3b251c50ad92c [id=27a0981b71084436994f1e9689afb145]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.518592 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8890a6a93e3d48dc96955f1b51e78603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.519589 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e9b6101e3447fb3c24b3917aa4872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.521303 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd3a1b1d3e44feda6dbaa8556e377fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.524986 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d409409813420689134773c627833b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.528967 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daac99da5d10475c9de263bef27baeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.529421 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9558dfe563d94482bf496e19b8a80edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.529924 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed134863bf642e79c26e5bf75b248b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.560549 1160 ts_tablet_manager.cc:1918] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.560832 1160 log.cc:1198] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4f5a90e65e80494a8b451f2bc6baa2b1
I20250109 20:52:46.561321 1160 ts_tablet_manager.cc:1939] T 4f5a90e65e80494a8b451f2bc6baa2b1 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:46.564987 1268 raft_consensus.cc:491] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:46.566610 747 tablet_service.cc:1467] Processing CreateTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc (DEFAULT_TABLE table=default.Table_a137a6b3e0614575b0a4f169e928ce95 [id=7adda20a8381463a9f6decd4c7cd045d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.567339 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9558dfe563d94482bf496e19b8a80edc (DEFAULT_TABLE table=default.Table_9c94d3852ae945d0b0dd5798dac99fba [id=095b22eda1d34de4b3394189471c53c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:46.568881 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd3d0bfb39db40758e82be511d033b4c consensus state has no leader
I20250109 20:52:46.569630 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_01fe90e030ba4f5cac35ad1e4797122d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.570149 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01fe90e030ba4f5cac35ad1e4797122d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.571698 452 master_service.cc:946] ReplaceTablet: received request to replace tablet ffd6ce78438841daa663b6344f062019 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.572012 1268 raft_consensus.cc:513] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.572839 1268 leader_election.cc:290] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:46.573132 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_06b1c0265e874ce39301f61e327e20ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.573704 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06b1c0265e874ce39301f61e327e20ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.573966 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd3a1b1d3e44feda6dbaa8556e377fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.575356 1166 log.cc:1198] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/32eb4dfcf96c4a05a55082e90ae04467
I20250109 20:52:46.576642 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0c3c6e0557ae46a0a8854d04f3d303e0" } modify_external_catalogs: true
I20250109 20:52:46.576673 753 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 (DEFAULT_TABLE table=default.Table_70499a7567fa4d6498705309cde3b257 [id=3ae741f5fe4343188b6f0d10b2a09b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.576797 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0c3c6e0557ae46a0a8854d04f3d303e0" } modify_external_catalogs: true
I20250109 20:52:46.577111 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4f047e8fa34ca79e78915c61a8ba12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.577703 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_756f5f6ef85d468586a5fb77b0cc441d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.578117 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_756f5f6ef85d468586a5fb77b0cc441d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.578883 1166 ts_tablet_manager.cc:1939] T 32eb4dfcf96c4a05a55082e90ae04467 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.580380 1166 tablet_replica.cc:331] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:46.580569 1166 raft_consensus.cc:2238] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.580739 1166 raft_consensus.cc:2267] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.581163 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9558dfe563d94482bf496e19b8a80edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.581249 1166 ts_tablet_manager.cc:1905] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.582098 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_84128457bfd1467cb5927699063cfe0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.582492 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84128457bfd1467cb5927699063cfe0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.583467 745 tablet_service.cc:1467] Processing CreateTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 (DEFAULT_TABLE table=default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.583878 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8890a6a93e3d48dc96955f1b51e78603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.586206 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7ed9e8c921491c9d911d662f500451" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:46.586418 1054 raft_consensus.cc:2463] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:46.586431 741 tablet_service.cc:1467] Processing CreateTablet for tablet 57e56f30e6f3450a8426cd40db89e009 (DEFAULT_TABLE table=default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.586719 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e56f30e6f3450a8426cd40db89e009. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.586865 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 32eb4dfcf96c4a05a55082e90ae04467 (table default.Table_384c816ede6b4c558524a02495bb3dc0 [id=756ad36069be45acaadb834d6ad41d05]) successfully deleted
W20250109 20:52:46.587133 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4d8e7081cac440389bec5dfac2391beb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d8e7081cac440389bec5dfac2391beb already in progress: creating tablet
I20250109 20:52:46.585625 1023 tablet_service.cc:1467] Processing CreateTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 (DEFAULT_TABLE table=default.Table_0bef1709ef9c47738d52c3d41d4f6819 [id=65098b4e94ce440494b2d5682132ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:46.587473 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e4c367a8ca184dfa9e4b11094ba76140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4c367a8ca184dfa9e4b11094ba76140 already in progress: creating tablet
W20250109 20:52:46.587667 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f5a5815d9ef54e3e959f151ed08ae812 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5a5815d9ef54e3e959f151ed08ae812 already in progress: creating tablet
W20250109 20:52:46.588167 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ffd6ce78438841daa663b6344f062019 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffd6ce78438841daa663b6344f062019 already in progress: creating tablet
I20250109 20:52:46.589607 713 leader_election.cc:304] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:46.589478 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 93d409409813420689134773c627833b (DEFAULT_TABLE table=default.Table_1ee95fba580446b8926cbc779fca720f [id=a4a7c86d0df2412a83f12157c967aae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.589901 758 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed134863bf642e79c26e5bf75b248b0 (DEFAULT_TABLE table=default.Table_0a104f68c5934305bc28de2aa457e751 [id=46c011650d964e26b40be8a854e1f8c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.590304 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc with delete_type TABLET_DATA_DELETED (replaced by tablet cd23012af5574f37944f9e26c07d05d2) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.590462 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed134863bf642e79c26e5bf75b248b0 (DEFAULT_TABLE table=default.Table_0a104f68c5934305bc28de2aa457e751 [id=46c011650d964e26b40be8a854e1f8c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.590943 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed134863bf642e79c26e5bf75b248b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.591333 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb0d889cb1114edc8b799b41b23a54a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.591822 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d409409813420689134773c627833b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.592298 740 tablet_service.cc:1467] Processing CreateTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 (DEFAULT_TABLE table=default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.592653 1033 tablet_service.cc:1467] Processing CreateTablet for tablet daac99da5d10475c9de263bef27baeb7 (DEFAULT_TABLE table=default.Table_d70feebb4775482eb8b3b251c50ad92c [id=27a0981b71084436994f1e9689afb145]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.593086 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 34f215eee55d4e6a97a384df6ff9d596 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.593508 1034 tablet_service.cc:1467] Processing CreateTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e (DEFAULT_TABLE table=default.Table_14ad6cd05135486e87607f62e9dd62c4 [id=eccfc40a5a774588b25444c9d6f1507d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.593801 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed134863bf642e79c26e5bf75b248b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.596253 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 57e56f30e6f3450a8426cd40db89e009 (DEFAULT_TABLE table=default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.598538 748 tablet_service.cc:1467] Processing CreateTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 (DEFAULT_TABLE table=default.Table_0bef1709ef9c47738d52c3d41d4f6819 [id=65098b4e94ce440494b2d5682132ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.600484 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_aaff98763d0f4eff96db3f5d2d85ba2a" } modify_external_catalogs: true
I20250109 20:52:46.600577 750 tablet_service.cc:1467] Processing CreateTablet for tablet 93d409409813420689134773c627833b (DEFAULT_TABLE table=default.Table_1ee95fba580446b8926cbc779fca720f [id=a4a7c86d0df2412a83f12157c967aae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.600641 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_aaff98763d0f4eff96db3f5d2d85ba2a" } modify_external_catalogs: true
I20250109 20:52:46.600749 1268 raft_consensus.cc:2798] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:46.600831 1268 raft_consensus.cc:491] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:46.601756 1268 raft_consensus.cc:3054] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:46.599684 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.604048 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_17129d3d3fd24e9386fd3228f1e15342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.604568 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17129d3d3fd24e9386fd3228f1e15342 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.606329 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728e9b6101e3447fb3c24b3917aa4872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.607508 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d409409813420689134773c627833b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.607609 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 34f215eee55d4e6a97a384df6ff9d596
I20250109 20:52:46.607992 453 catalog_manager.cc:6596] ReplaceTablet: tablet 34f215eee55d4e6a97a384df6ff9d596 deleted and replaced by tablet e16a1d9a5b374b8dbf725eb32ffadb9e
I20250109 20:52:46.608692 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2b47656fe1ae4c9ba4138bcd37c0e6db
I20250109 20:52:46.608760 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d09daa2eed41c3952fa21aa2cdd09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.609329 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e56f30e6f3450a8426cd40db89e009. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.610819 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 with delete_type TABLET_DATA_DELETED (replaced by tablet e16a1d9a5b374b8dbf725eb32ffadb9e) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.610997 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daac99da5d10475c9de263bef27baeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.611299 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.612041 742 tablet_service.cc:1467] Processing CreateTablet for tablet daac99da5d10475c9de263bef27baeb7 (DEFAULT_TABLE table=default.Table_d70feebb4775482eb8b3b251c50ad92c [id=27a0981b71084436994f1e9689afb145]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.612999 746 tablet_service.cc:1467] Processing CreateTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e (DEFAULT_TABLE table=default.Table_14ad6cd05135486e87607f62e9dd62c4 [id=eccfc40a5a774588b25444c9d6f1507d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.620050 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.621822 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7c349e16f1b547c2b54536d168ff9e4a" } modify_external_catalogs: true
I20250109 20:52:46.622009 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7c349e16f1b547c2b54536d168ff9e4a" } modify_external_catalogs: true
I20250109 20:52:46.622951 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4c30a9425b844ee187a29bc12cd2704e" } modify_external_catalogs: true
I20250109 20:52:46.623101 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4c30a9425b844ee187a29bc12cd2704e" } modify_external_catalogs: true
I20250109 20:52:46.623180 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daac99da5d10475c9de263bef27baeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.624320 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c4b9fe08755e4cb1b56978cbda1f2025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.624908 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4b9fe08755e4cb1b56978cbda1f2025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.626783 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e8b35b282b6f4dc9bbb6038afd135892" } new_table_name: "default.Table_214cf06447eb4076abbed4a92f99fc79" modify_external_catalogs: true
I20250109 20:52:46.633091 1268 raft_consensus.cc:513] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.634788 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ff4f020c5ba643aeb11f8e08d748be46
I20250109 20:52:46.635142 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7ed9e8c921491c9d911d662f500451" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:46.635399 1054 raft_consensus.cc:3054] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:46.638106 1176 tablet_bootstrap.cc:492] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:46.638273 1176 ts_tablet_manager.cc:1397] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.166s user 0.006s sys 0.024s
I20250109 20:52:46.639314 1054 raft_consensus.cc:2463] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:46.639889 713 leader_election.cc:304] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:46.641203 1166 ts_tablet_manager.cc:1918] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.641352 1166 log.cc:1198] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/35d7cead4f4149ae8de48eee2cfd8679
I20250109 20:52:46.642144 1166 ts_tablet_manager.cc:1939] T 35d7cead4f4149ae8de48eee2cfd8679 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.643621 1166 tablet_replica.cc:331] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:46.643802 1166 raft_consensus.cc:2238] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.643977 1166 raft_consensus.cc:2267] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.644466 1166 ts_tablet_manager.cc:1905] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.648799 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c4e1ecf827064d4198b94eee6e6b4120"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.649398 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4e1ecf827064d4198b94eee6e6b4120 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.650429 1176 raft_consensus.cc:357] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.650697 1176 raft_consensus.cc:383] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.650813 1176 raft_consensus.cc:738] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.651104 1176 consensus_queue.cc:260] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.653028 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.653105 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5a6fe1039e374d2bb76ad9117530c055"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.653602 1176 ts_tablet_manager.cc:1428] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
W20250109 20:52:46.653594 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a6fe1039e374d2bb76ad9117530c055 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.653993 1176 tablet_bootstrap.cc:492] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:46.656595 1176 tablet_bootstrap.cc:654] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.657002 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0bef1709ef9c47738d52c3d41d4f6819" } modify_external_catalogs: true
I20250109 20:52:46.657187 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0bef1709ef9c47738d52c3d41d4f6819" } modify_external_catalogs: true
I20250109 20:52:46.658092 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8b85547ea1c642a693a94f23f77df03f" } new_table_name: "default.Table_4ddee41d2e28400c93caa1462c016e5b" modify_external_catalogs: true
W20250109 20:52:46.662302 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3000b819a4cc4960909b98febd400631 consensus state has no leader
I20250109 20:52:46.664865 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 35d7cead4f4149ae8de48eee2cfd8679 (table default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
I20250109 20:52:46.666685 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_84128457bfd1467cb5927699063cfe0c" } new_table_name: "default.Table_4887e41bfb6f4a4aa53cb437224c8e1c" modify_external_catalogs: true
I20250109 20:52:46.672174 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet ed4866f6007d411da5cf4832fa92ece9 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]). Expected version 2 got 0
I20250109 20:52:46.672236 1268 leader_election.cc:290] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:46.672422 1268 raft_consensus.cc:2798] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
W20250109 20:52:46.673998 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c5e9b31b27949c2948cb64b6124ee72 consensus state has no leader
I20250109 20:52:46.674067 1268 raft_consensus.cc:695] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:46.674355 1268 consensus_queue.cc:237] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.678646 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb0d889cb1114edc8b799b41b23a54a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.687283 758 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.687808 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a8da9781ab32429297d0c875c1971501
I20250109 20:52:46.687896 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bb0d889cb1114edc8b799b41b23a54a6
W20250109 20:52:46.688095 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72cc0bbbb5584f14ba4d6c62211a12c5 has no consensus state
I20250109 20:52:46.688503 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet deed945f2ecf4290ad002662c3f66030
I20250109 20:52:46.694371 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 with delete_type TABLET_DATA_DELETED (replaced by tablet e16a1d9a5b374b8dbf725eb32ffadb9e) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.561544 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4f047e8fa34ca79e78915c61a8ba12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.573729 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7ed9e8c921491c9d911d662f500451" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:46.701231 918 raft_consensus.cc:2463] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:46.614799 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 with delete_type TABLET_DATA_DELETED (replaced by tablet e16a1d9a5b374b8dbf725eb32ffadb9e) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.634441 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7ed9e8c921491c9d911d662f500451" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:46.704949 919 raft_consensus.cc:3054] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:46.708029 919 raft_consensus.cc:2463] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:46.708925 455 master_service.cc:946] ReplaceTablet: received request to replace tablet c40e5019321c497b814a281e0ab1a7f7 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.662163 896 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.663861 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.709425 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb0d889cb1114edc8b799b41b23a54a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.709676 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6db1fc2ef56942dc9e962b951f82a995" } new_table_name: "default.Table_df185d0c630644438a9e8cc5fe14446b" modify_external_catalogs: true
I20250109 20:52:46.711061 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 7da1f3cafffe4f7ca39624b43831689e (DEFAULT_TABLE table=default.Table_4f4640c9222649fc957636279628a391 [id=cc1214f3eaf54bf0ba8167dbd8a4e634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.711125 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ca816507d0c047668a51f6e7ebce18cb (DEFAULT_TABLE table=default.Table_7d999163ab094d83933d7da655cb380a [id=b4e108f1f95045159449a7bde6731cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.711465 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca816507d0c047668a51f6e7ebce18cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.711473 1023 tablet_service.cc:1467] Processing CreateTablet for tablet cd23012af5574f37944f9e26c07d05d2 (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.711887 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd23012af5574f37944f9e26c07d05d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.712888 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e56f30e6f3450a8426cd40db89e009. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.712955 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da1f3cafffe4f7ca39624b43831689e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.720548 1166 ts_tablet_manager.cc:1918] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.667634 886 tablet_service.cc:1467] Processing CreateTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e (DEFAULT_TABLE table=default.Table_14ad6cd05135486e87607f62e9dd62c4 [id=eccfc40a5a774588b25444c9d6f1507d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.673583 1165 tablet_bootstrap.cc:492] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:46.679420 889 tablet_service.cc:1467] Processing CreateTablet for tablet ca816507d0c047668a51f6e7ebce18cb (DEFAULT_TABLE table=default.Table_7d999163ab094d83933d7da655cb380a [id=b4e108f1f95045159449a7bde6731cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.721252 1165 ts_tablet_manager.cc:1397] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.450s user 0.005s sys 0.000s
I20250109 20:52:46.721376 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d09daa2eed41c3952fa21aa2cdd09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.721625 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca816507d0c047668a51f6e7ebce18cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.722002 1165 raft_consensus.cc:357] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.722172 1165 raft_consensus.cc:383] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.722249 1165 raft_consensus.cc:738] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.722484 1165 consensus_queue.cc:260] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.723043 1165 ts_tablet_manager.cc:1428] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:46.723371 1165 tablet_bootstrap.cc:492] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:46.728415 1165 tablet_bootstrap.cc:654] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.679841 895 tablet_service.cc:1467] Processing CreateTablet for tablet 7da1f3cafffe4f7ca39624b43831689e (DEFAULT_TABLE table=default.Table_4f4640c9222649fc957636279628a391 [id=cc1214f3eaf54bf0ba8167dbd8a4e634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.731328 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da1f3cafffe4f7ca39624b43831689e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.681571 893 tablet_service.cc:1467] Processing CreateTablet for tablet cd23012af5574f37944f9e26c07d05d2 (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.733023 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd23012af5574f37944f9e26c07d05d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.734513 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4f5a90e65e80494a8b451f2bc6baa2b1 (table default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]) successfully deleted
W20250109 20:52:46.736382 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a5a6dbb823e4bd3afeb157b3c5771ba already in progress: creating tablet
I20250109 20:52:46.736862 1160 tablet_replica.cc:331] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:46.737119 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 97205d09de264adb9d19cb32546b0942 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.737195 1160 raft_consensus.cc:2238] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.737535 1160 raft_consensus.cc:2267] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.738360 1160 ts_tablet_manager.cc:1905] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.740861 744 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc with delete_type TABLET_DATA_DELETED (replaced by tablet cd23012af5574f37944f9e26c07d05d2) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.743010 739 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.743228 758 tablet_service.cc:1467] Processing CreateTablet for tablet 7da1f3cafffe4f7ca39624b43831689e (DEFAULT_TABLE table=default.Table_4f4640c9222649fc957636279628a391 [id=cc1214f3eaf54bf0ba8167dbd8a4e634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.743556 1158 tablet_bootstrap.cc:492] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:46.743927 751 tablet_service.cc:1514] Processing DeleteTablet for tablet deed945f2ecf4290ad002662c3f66030 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.744027 1158 ts_tablet_manager.cc:1397] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.425s user 0.068s sys 0.097s
I20250109 20:52:46.743674 749 tablet_service.cc:1467] Processing CreateTablet for tablet ca816507d0c047668a51f6e7ebce18cb (DEFAULT_TABLE table=default.Table_7d999163ab094d83933d7da655cb380a [id=b4e108f1f95045159449a7bde6731cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.744541 753 tablet_service.cc:1514] Processing DeleteTablet for tablet a8da9781ab32429297d0c875c1971501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.746218 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d09daa2eed41c3952fa21aa2cdd09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.746768 747 tablet_service.cc:1467] Processing CreateTablet for tablet cd23012af5574f37944f9e26c07d05d2 (DEFAULT_TABLE table=default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.695305 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc with delete_type TABLET_DATA_DELETED (replaced by tablet cd23012af5574f37944f9e26c07d05d2) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.696491 885 tablet_service.cc:1514] Processing DeleteTablet for tablet deed945f2ecf4290ad002662c3f66030 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.752821 1158 raft_consensus.cc:357] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.696683 888 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.696900 897 tablet_service.cc:1514] Processing DeleteTablet for tablet a8da9781ab32429297d0c875c1971501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:46.744421 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 60183.8us
W20250109 20:52:46.755255 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c03524194a7b4e34a20424091e308ac4 consensus state has no leader
I20250109 20:52:46.757503 1158 raft_consensus.cc:383] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.757316 1385 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:46.760475 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 5757b98100804c688f67e7111db2b622 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.760838 1158 raft_consensus.cc:738] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.761133 1158 consensus_queue.cc:260] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.762034 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca816507d0c047668a51f6e7ebce18cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.732044 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet a8da9781ab32429297d0c875c1971501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.762698 1166 log.cc:1198] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/66c6016d88b54eb390f880945ddb1ea6
I20250109 20:52:46.762831 1158 ts_tablet_manager.cc:1428] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250109 20:52:46.763171 1158 tablet_bootstrap.cc:492] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:46.763411 1166 ts_tablet_manager.cc:1939] T 66c6016d88b54eb390f880945ddb1ea6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.732336 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.732647 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet deed945f2ecf4290ad002662c3f66030 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.740844 1187 raft_consensus.cc:491] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:46.764411 1187 raft_consensus.cc:513] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.764560 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 66c6016d88b54eb390f880945ddb1ea6 (table default.Table_f9f28decdfb64455809f7b283552a4a5 [id=ae7fb9326c2c49a593d1111d45ea424f]) successfully deleted
I20250109 20:52:46.752768 1176 tablet_bootstrap.cc:492] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:46.764851 1176 ts_tablet_manager.cc:1397] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.111s user 0.001s sys 0.003s
I20250109 20:52:46.765532 1176 raft_consensus.cc:357] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.765748 1176 raft_consensus.cc:383] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.765846 1176 raft_consensus.cc:738] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.766052 1176 consensus_queue.cc:260] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.767064 1187 leader_election.cc:290] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:46.767190 1187 raft_consensus.cc:491] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:46.767297 1187 raft_consensus.cc:513] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:46.767745 1187 leader_election.cc:290] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:46.767832 1157 ts_tablet_manager.cc:1918] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.768002 1157 log.cc:1198] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/97205d09de264adb9d19cb32546b0942
I20250109 20:52:46.769129 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd23012af5574f37944f9e26c07d05d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.769405 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a54393ff3264c52bfdeee2cb9046828" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:46.770633 990 leader_election.cc:343] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.771265 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a54393ff3264c52bfdeee2cb9046828" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:46.771955 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acadf60d566407aabc1a2395d1d235f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:46.772550 989 leader_election.cc:343] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.772687 989 leader_election.cc:304] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:46.773075 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acadf60d566407aabc1a2395d1d235f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:46.773751 990 leader_election.cc:343] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:46.773945 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4015c0e66dfd47bfb9855f82044f01e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4015c0e66dfd47bfb9855f82044f01e7 already in progress: creating tablet
W20250109 20:52:46.774134 989 leader_election.cc:343] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:46.774250 989 leader_election.cc:304] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:46.774641 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da1f3cafffe4f7ca39624b43831689e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.774681 1187 raft_consensus.cc:2743] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:46.774855 1187 raft_consensus.cc:2743] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:46.776417 1158 tablet_bootstrap.cc:654] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.778579 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5757b98100804c688f67e7111db2b622
I20250109 20:52:46.779109 446 catalog_manager.cc:6596] ReplaceTablet: tablet 5757b98100804c688f67e7111db2b622 deleted and replaced by tablet 4f8419d8cc7e475ea43dd263cae87aca
I20250109 20:52:46.779914 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757b98100804c688f67e7111db2b622 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f8419d8cc7e475ea43dd263cae87aca) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.781993 1157 ts_tablet_manager.cc:1939] T 97205d09de264adb9d19cb32546b0942 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:46.783723 1296 fs_manager.cc:730] Time spent opening block manager: real 0.328s user 0.020s sys 0.040s
I20250109 20:52:46.784391 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757b98100804c688f67e7111db2b622 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f8419d8cc7e475ea43dd263cae87aca) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.783959 1296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
uuid: "36a29ab2002d4eb389cd11778988e1e9"
format_stamp: "Formatted at 2025-01-09 20:52:40 on dist-test-slave-612g"
I20250109 20:52:46.785509 446 catalog_manager.cc:5526] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:46.785688 1296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:52:46.786034 446 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170]). Expected version 2 got 0
I20250109 20:52:46.789994 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757b98100804c688f67e7111db2b622 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f8419d8cc7e475ea43dd263cae87aca) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.790408 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6cca94c4ba974861b95038c43ebf833a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.790795 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_10e1ff86f4e8433f8f5174dd6d3f19fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.791008 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cca94c4ba974861b95038c43ebf833a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:46.791244 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10e1ff86f4e8433f8f5174dd6d3f19fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.794310 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_609b1c68b73f4e99ad6afa14ee14ad50" } new_table_name: "default.Table_bc5a2395f34546189d1ebbeeecd1538e" modify_external_catalogs: true
I20250109 20:52:46.794466 1166 tablet_replica.cc:331] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:46.805485 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet ed4866f6007d411da5cf4832fa92ece9 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]). Expected version 2 got 0
I20250109 20:52:46.808422 1160 ts_tablet_manager.cc:1918] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.808635 1160 log.cc:1198] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/175bf6c335a8405c97b296cf4718610b
I20250109 20:52:46.809147 1160 ts_tablet_manager.cc:1939] T 175bf6c335a8405c97b296cf4718610b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:46.809715 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 175bf6c335a8405c97b296cf4718610b (table default.Table_5d1e5a64991f49a48a7b4286fc1ffe5f [id=a6f0a5fcb43d4509a188259adfdd0192]) successfully deleted
W20250109 20:52:46.810211 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 41de7acfa7f941caa94bdbb888cf841e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41de7acfa7f941caa94bdbb888cf841e already in progress: creating tablet
W20250109 20:52:46.810421 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet aa7d7695ff5946abb6ee576431d2e87f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa7d7695ff5946abb6ee576431d2e87f already in progress: creating tablet
W20250109 20:52:46.810632 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2d96a569e6be4e01a26918d115cfbcf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d96a569e6be4e01a26918d115cfbcf9 already in progress: creating tablet
I20250109 20:52:46.810693 1160 tablet_replica.cc:331] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:46.810886 1160 raft_consensus.cc:2238] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.811064 1160 raft_consensus.cc:2267] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:46.811107 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6acca65ecee94f8cbe3f2b89e744aa6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6acca65ecee94f8cbe3f2b89e744aa6c already in progress: creating tablet
W20250109 20:52:46.811297 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 49c6f1833ff1453f90a6ec6435248a02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49c6f1833ff1453f90a6ec6435248a02 already in progress: creating tablet
W20250109 20:52:46.811488 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 34f215eee55d4e6a97a384df6ff9d596 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34f215eee55d4e6a97a384df6ff9d596 already in progress: creating tablet
W20250109 20:52:46.811693 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ff4f020c5ba643aeb11f8e08d748be46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4f020c5ba643aeb11f8e08d748be46 already in progress: creating tablet
I20250109 20:52:46.811852 1160 ts_tablet_manager.cc:1905] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:46.811884 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b47656fe1ae4c9ba4138bcd37c0e6db already in progress: creating tablet
I20250109 20:52:46.817226 1166 raft_consensus.cc:2238] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.817438 1166 raft_consensus.cc:2267] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.817375 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ce7eb75c66cc418e8f6b1729d7d51e57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.817894 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce7eb75c66cc418e8f6b1729d7d51e57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.818261 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_65ca714c159440f4a1cc204e1c50d821"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.818666 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65ca714c159440f4a1cc204e1c50d821 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.822427 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2e446345a181499d86750bfaca472d62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.822945 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e446345a181499d86750bfaca472d62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.823485 890 tablet_service.cc:1467] Processing CreateTablet for tablet 04e46f4312844b6eb29d6718f4800f98 (DEFAULT_TABLE table=default.Table_c4b9fe08755e4cb1b56978cbda1f2025 [id=4394b742727948f99337b3aa45e81de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.823917 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e46f4312844b6eb29d6718f4800f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.824817 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 04e46f4312844b6eb29d6718f4800f98 (DEFAULT_TABLE table=default.Table_c4b9fe08755e4cb1b56978cbda1f2025 [id=4394b742727948f99337b3aa45e81de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.824965 1166 ts_tablet_manager.cc:1905] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.825588 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e46f4312844b6eb29d6718f4800f98. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.826802 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b31b9785ee21418ba13b426ba94cfa9b consensus state has no leader
I20250109 20:52:46.841126 1176 ts_tablet_manager.cc:1428] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.076s user 0.000s sys 0.014s
I20250109 20:52:46.843065 1176 tablet_bootstrap.cc:492] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:46.843149 1160 ts_tablet_manager.cc:1918] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.843283 1160 log.cc:1198] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/97205d09de264adb9d19cb32546b0942
I20250109 20:52:46.847303 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 97205d09de264adb9d19cb32546b0942 (table default.Table_1bca442744ee4e04a46973fb21434cae [id=45220eb0588a4c62838a951101e7e04f]) successfully deleted
W20250109 20:52:46.848474 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet aa7d7695ff5946abb6ee576431d2e87f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa7d7695ff5946abb6ee576431d2e87f already in progress: creating tablet
I20250109 20:52:46.849493 1160 ts_tablet_manager.cc:1939] T 97205d09de264adb9d19cb32546b0942 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:46.848659 1165 tablet_bootstrap.cc:492] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:46.849788 1165 ts_tablet_manager.cc:1397] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.017s
I20250109 20:52:46.850610 745 tablet_service.cc:1467] Processing CreateTablet for tablet 04e46f4312844b6eb29d6718f4800f98 (DEFAULT_TABLE table=default.Table_c4b9fe08755e4cb1b56978cbda1f2025 [id=4394b742727948f99337b3aa45e81de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.850728 1165 raft_consensus.cc:357] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.850924 1165 raft_consensus.cc:383] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.850987 1165 raft_consensus.cc:738] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.851207 1165 consensus_queue.cc:260] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:46.852972 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 41de7acfa7f941caa94bdbb888cf841e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41de7acfa7f941caa94bdbb888cf841e already in progress: creating tablet
I20250109 20:52:46.853747 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 97205d09de264adb9d19cb32546b0942 (table default.Table_1bca442744ee4e04a46973fb21434cae [id=45220eb0588a4c62838a951101e7e04f]) successfully deleted
W20250109 20:52:46.853978 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet deed945f2ecf4290ad002662c3f66030 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deed945f2ecf4290ad002662c3f66030 already in progress: creating tablet
W20250109 20:52:46.854215 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet bb0d889cb1114edc8b799b41b23a54a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb0d889cb1114edc8b799b41b23a54a6 already in progress: creating tablet
W20250109 20:52:46.854409 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a8da9781ab32429297d0c875c1971501 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8da9781ab32429297d0c875c1971501 already in progress: creating tablet
W20250109 20:52:46.854629 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ed5ea5d588834f1b8b2b057ec324aadc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed5ea5d588834f1b8b2b057ec324aadc already in progress: creating tablet
W20250109 20:52:46.854856 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5757b98100804c688f67e7111db2b622 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5757b98100804c688f67e7111db2b622 already in progress: creating tablet
I20250109 20:52:46.855923 899 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.856246 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e46f4312844b6eb29d6718f4800f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.859655 1158 tablet_bootstrap.cc:492] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:46.860484 1158 ts_tablet_manager.cc:1397] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.097s user 0.008s sys 0.027s
I20250109 20:52:46.864521 1158 raft_consensus.cc:357] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:46.865053 1158 raft_consensus.cc:383] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:46.865136 1158 raft_consensus.cc:738] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:46.866240 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_642aeae240da4ea3a0fca064e8489134" } modify_external_catalogs: true
I20250109 20:52:46.866734 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c40e5019321c497b814a281e0ab1a7f7
I20250109 20:52:46.866834 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9cbf7236e62e4c22a28e6ccb34f76495"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:46.867071 455 catalog_manager.cc:6596] ReplaceTablet: tablet c40e5019321c497b814a281e0ab1a7f7 deleted and replaced by tablet f4eb5d2e7e68498bbfa47285babedaa6
W20250109 20:52:46.867431 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cbf7236e62e4c22a28e6ccb34f76495 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.868474 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_642aeae240da4ea3a0fca064e8489134" } modify_external_catalogs: true
I20250109 20:52:46.871344 741 tablet_service.cc:1467] Processing CreateTablet for tablet d845bac2539e478083bf7fe6cbc0c717 (DEFAULT_TABLE table=default.Table_17129d3d3fd24e9386fd3228f1e15342 [id=2509b1e547034a9ea53a95d04a14cf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.871440 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_56c8a1eb8409437c856c4af9ece36c61" } modify_external_catalogs: true
I20250109 20:52:46.871621 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_56c8a1eb8409437c856c4af9ece36c61" } modify_external_catalogs: true
I20250109 20:52:46.871724 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d845bac2539e478083bf7fe6cbc0c717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.872736 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e9297199b4064b5f889530ac2757c3ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.873214 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9297199b4064b5f889530ac2757c3ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.874528 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_13f472ba51244613ab6c1c0cc5e9f9e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.875010 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13f472ba51244613ab6c1c0cc5e9f9e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.876056 453 master_service.cc:946] ReplaceTablet: received request to replace tablet f5a5815d9ef54e3e959f151ed08ae812 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:46.876737 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cc174d6bd9f44eaf9f66e54693597601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc174d6bd9f44eaf9f66e54693597601 already in progress: creating tablet
I20250109 20:52:46.856072 1165 ts_tablet_manager.cc:1428] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:52:46.869302 892 tablet_service.cc:1467] Processing CreateTablet for tablet d845bac2539e478083bf7fe6cbc0c717 (DEFAULT_TABLE table=default.Table_17129d3d3fd24e9386fd3228f1e15342 [id=2509b1e547034a9ea53a95d04a14cf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.869758 895 tablet_service.cc:1467] Processing CreateTablet for tablet 14fb1017fe6c4f5abd640ad1f6214232 (DEFAULT_TABLE table=default.Table_5a6fe1039e374d2bb76ad9117530c055 [id=26365b65a557409abc651578b4458fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.877246 1165 tablet_bootstrap.cc:492] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:46.877314 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_af5240c88be241afba737b04efe6456e" } new_table_name: "default.Table_3585fc2fc449422880a7ebe625c22d09" modify_external_catalogs: true
I20250109 20:52:46.877411 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d845bac2539e478083bf7fe6cbc0c717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.869918 888 tablet_service.cc:1467] Processing CreateTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e (DEFAULT_TABLE table=default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.878314 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16a1d9a5b374b8dbf725eb32ffadb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.881291 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fb1017fe6c4f5abd640ad1f6214232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.870325 893 tablet_service.cc:1467] Processing CreateTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 (DEFAULT_TABLE table=default.Table_01fe90e030ba4f5cac35ad1e4797122d [id=facd2e83422c492aa4337beff9ceb548]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.870436 890 tablet_service.cc:1467] Processing CreateTablet for tablet 825220629faa4d2481e0db9fa226c306 (DEFAULT_TABLE table=default.Table_756f5f6ef85d468586a5fb77b0cc441d [id=f4c5a8b765814c37882dfdeaf55481e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.883992 1158 consensus_queue.cc:260] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:46.884804 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 49c6f1833ff1453f90a6ec6435248a02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49c6f1833ff1453f90a6ec6435248a02 already in progress: creating tablet
W20250109 20:52:46.885048 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6acca65ecee94f8cbe3f2b89e744aa6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6acca65ecee94f8cbe3f2b89e744aa6c already in progress: creating tablet
W20250109 20:52:46.885252 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2d96a569e6be4e01a26918d115cfbcf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d96a569e6be4e01a26918d115cfbcf9 already in progress: creating tablet
W20250109 20:52:46.885430 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b47656fe1ae4c9ba4138bcd37c0e6db already in progress: creating tablet
W20250109 20:52:46.885725 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ff4f020c5ba643aeb11f8e08d748be46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4f020c5ba643aeb11f8e08d748be46 already in progress: creating tablet
W20250109 20:52:46.885900 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 34f215eee55d4e6a97a384df6ff9d596 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34f215eee55d4e6a97a384df6ff9d596 already in progress: creating tablet
W20250109 20:52:46.886102 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ed5ea5d588834f1b8b2b057ec324aadc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed5ea5d588834f1b8b2b057ec324aadc already in progress: creating tablet
W20250109 20:52:46.886301 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet bb0d889cb1114edc8b799b41b23a54a6 because the tablet was not found. No further retry: Not found: Tablet not found: bb0d889cb1114edc8b799b41b23a54a6
W20250109 20:52:46.886528 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet deed945f2ecf4290ad002662c3f66030 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deed945f2ecf4290ad002662c3f66030 already in progress: creating tablet
W20250109 20:52:46.886716 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a8da9781ab32429297d0c875c1971501 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8da9781ab32429297d0c875c1971501 already in progress: creating tablet
W20250109 20:52:46.886958 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5757b98100804c688f67e7111db2b622 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5757b98100804c688f67e7111db2b622 already in progress: creating tablet
I20250109 20:52:46.887938 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ebeffe516c4c239ae2ee9040276e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.887987 1165 tablet_bootstrap.cc:654] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.888101 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_07f19c8f31bb48c598a7a2b8dc7187af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.888629 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07f19c8f31bb48c598a7a2b8dc7187af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.888487 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1b9aa886edd542fc8039fd921d1048c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:46.889067 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b9aa886edd542fc8039fd921d1048c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:46.889789 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9817e2e17d4d4c5ebcbf9ee51fae9654" } new_table_name: "default.Table_8d504623c1894cb8843ead0ee687ea13" modify_external_catalogs: true
I20250109 20:52:46.870671 887 tablet_service.cc:1467] Processing CreateTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 (DEFAULT_TABLE table=default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.890268 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825220629faa4d2481e0db9fa226c306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.890633 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72cc0bbbb5584f14ba4d6c62211a12c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.890972 1027 tablet_service.cc:1467] Processing CreateTablet for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (DEFAULT_TABLE table=default.Table_06b1c0265e874ce39301f61e327e20ff [id=fef90e9d048840acaf567c09af9ac9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.891394 1032 tablet_service.cc:1467] Processing CreateTablet for tablet d845bac2539e478083bf7fe6cbc0c717 (DEFAULT_TABLE table=default.Table_17129d3d3fd24e9386fd3228f1e15342 [id=2509b1e547034a9ea53a95d04a14cf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.891844 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.892139 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 14fb1017fe6c4f5abd640ad1f6214232 (DEFAULT_TABLE table=default.Table_5a6fe1039e374d2bb76ad9117530c055 [id=26365b65a557409abc651578b4458fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.892382 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 (DEFAULT_TABLE table=default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.892653 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 (DEFAULT_TABLE table=default.Table_01fe90e030ba4f5cac35ad1e4797122d [id=facd2e83422c492aa4337beff9ceb548]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.871192 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c40e5019321c497b814a281e0ab1a7f7 with delete_type TABLET_DATA_DELETED (replaced by tablet f4eb5d2e7e68498bbfa47285babedaa6) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.870930 886 tablet_service.cc:1467] Processing CreateTablet for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (DEFAULT_TABLE table=default.Table_06b1c0265e874ce39301f61e327e20ff [id=fef90e9d048840acaf567c09af9ac9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.893360 742 tablet_service.cc:1467] Processing CreateTablet for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (DEFAULT_TABLE table=default.Table_06b1c0265e874ce39301f61e327e20ff [id=fef90e9d048840acaf567c09af9ac9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.893942 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 825220629faa4d2481e0db9fa226c306 (DEFAULT_TABLE table=default.Table_756f5f6ef85d468586a5fb77b0cc441d [id=f4c5a8b765814c37882dfdeaf55481e2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:46.894464 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c40e5019321c497b814a281e0ab1a7f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c40e5019321c497b814a281e0ab1a7f7 already in progress: creating tablet
I20250109 20:52:46.894800 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc32ce3dff0f4d18bb17c6784b0bab41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.897946 754 tablet_service.cc:1514] Processing DeleteTablet for tablet c40e5019321c497b814a281e0ab1a7f7 with delete_type TABLET_DATA_DELETED (replaced by tablet f4eb5d2e7e68498bbfa47285babedaa6) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.898301 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 with delete_type TABLET_DATA_DELETED (replaced by tablet cba0ffb1b394431ba30918911cdee7b7) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.898705 744 tablet_service.cc:1467] Processing CreateTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e (DEFAULT_TABLE table=default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.899000 1158 ts_tablet_manager.cc:1428] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20250109 20:52:46.899076 753 tablet_service.cc:1467] Processing CreateTablet for tablet 14fb1017fe6c4f5abd640ad1f6214232 (DEFAULT_TABLE table=default.Table_5a6fe1039e374d2bb76ad9117530c055 [id=26365b65a557409abc651578b4458fd4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:46.900336 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c40e5019321c497b814a281e0ab1a7f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c40e5019321c497b814a281e0ab1a7f7 already in progress: creating tablet
I20250109 20:52:46.900811 740 tablet_service.cc:1467] Processing CreateTablet for tablet 825220629faa4d2481e0db9fa226c306 (DEFAULT_TABLE table=default.Table_756f5f6ef85d468586a5fb77b0cc441d [id=f4c5a8b765814c37882dfdeaf55481e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.904812 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc32ce3dff0f4d18bb17c6784b0bab41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.905319 750 tablet_service.cc:1467] Processing CreateTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 (DEFAULT_TABLE table=default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.906634 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e6d0cb94669f451389e1f2b2581c63b6" } modify_external_catalogs: true
I20250109 20:52:46.906826 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e6d0cb94669f451389e1f2b2581c63b6" } modify_external_catalogs: true
I20250109 20:52:46.907219 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet fbde2eecdaf4470dbc74b0c59f31643c
I20250109 20:52:46.907933 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_579b6c1d61dd4e80af5ea30b4e06dc35" } modify_external_catalogs: true
I20250109 20:52:46.908108 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_579b6c1d61dd4e80af5ea30b4e06dc35" } modify_external_catalogs: true
I20250109 20:52:46.908166 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 540531463c184b9abf226d7487939e60
I20250109 20:52:46.908797 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee" } modify_external_catalogs: true
I20250109 20:52:46.908990 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee" } modify_external_catalogs: true
I20250109 20:52:46.897868 751 tablet_service.cc:1467] Processing CreateTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 (DEFAULT_TABLE table=default.Table_01fe90e030ba4f5cac35ad1e4797122d [id=facd2e83422c492aa4337beff9ceb548]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.915617 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bc5a2395f34546189d1ebbeeecd1538e" } new_table_name: "default.Table_e46d69f1ef0344d591706192682b73c9" modify_external_catalogs: true
I20250109 20:52:46.916990 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16a1d9a5b374b8dbf725eb32ffadb9e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:46.918828 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be44ffc9e371452f8d5f4174fbfa752a consensus state has no leader
I20250109 20:52:46.923755 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fb1017fe6c4f5abd640ad1f6214232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.924868 1157 tablet_replica.cc:331] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:46.925045 1157 raft_consensus.cc:2238] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.925182 1157 raft_consensus.cc:2267] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.925791 1157 ts_tablet_manager.cc:1905] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.930936 1016 tablet_service.cc:1467] Processing CreateTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e (DEFAULT_TABLE table=default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:46.931350 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc32ce3dff0f4d18bb17c6784b0bab41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.931876 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet c40e5019321c497b814a281e0ab1a7f7 with delete_type TABLET_DATA_DELETED (replaced by tablet f4eb5d2e7e68498bbfa47285babedaa6) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.932132 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16a1d9a5b374b8dbf725eb32ffadb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.932348 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14fb1017fe6c4f5abd640ad1f6214232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.933071 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ebeffe516c4c239ae2ee9040276e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.935560 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72cc0bbbb5584f14ba4d6c62211a12c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.935895 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825220629faa4d2481e0db9fa226c306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.936185 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 540531463c184b9abf226d7487939e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.936300 891 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.936331 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.936486 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 540531463c184b9abf226d7487939e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.936609 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 3000b819a4cc4960909b98febd400631 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:46.936993 1158 tablet_bootstrap.cc:492] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:46.936661 899 tablet_service.cc:1514] Processing DeleteTablet for tablet cc174d6bd9f44eaf9f66e54693597601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:46.937821 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet fbde2eecdaf4470dbc74b0c59f31643c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbde2eecdaf4470dbc74b0c59f31643c already in progress: creating tablet
W20250109 20:52:46.938057 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 540531463c184b9abf226d7487939e60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 540531463c184b9abf226d7487939e60 already in progress: creating tablet
I20250109 20:52:46.938381 1160 tablet_replica.cc:331] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:46.938534 1160 raft_consensus.cc:2238] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.938683 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825220629faa4d2481e0db9fa226c306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.938696 1160 raft_consensus.cc:2267] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:46.939426 1160 ts_tablet_manager.cc:1905] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:46.939612 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ebeffe516c4c239ae2ee9040276e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.940603 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72cc0bbbb5584f14ba4d6c62211a12c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.941284 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d845bac2539e478083bf7fe6cbc0c717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:46.947458 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 72a455033b674e5a94fc37d0275eb313
W20250109 20:52:46.947965 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 219a4b508050490cb3f1ee418e0edbf8 consensus state has no leader
I20250109 20:52:46.948259 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a455033b674e5a94fc37d0275eb313 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:46.948307 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet f4eb5d2e7e68498bbfa47285babedaa6
I20250109 20:52:46.951871 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a455033b674e5a94fc37d0275eb313 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:46.953282 1158 tablet_bootstrap.cc:654] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:46.953814 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 268d9969a46544d9954f71c77170e76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.955866 746 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.957352 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 540531463c184b9abf226d7487939e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:46.958252 1160 ts_tablet_manager.cc:1918] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.958390 1160 log.cc:1198] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/cc174d6bd9f44eaf9f66e54693597601
I20250109 20:52:46.959369 1160 ts_tablet_manager.cc:1939] T cc174d6bd9f44eaf9f66e54693597601 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:46.960512 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet cc174d6bd9f44eaf9f66e54693597601 (table default.Table_c576a7a39c174fe6ac5c351e1b4f80d3 [id=37a12f94e9c54332af6b774b56b13666]) successfully deleted
I20250109 20:52:46.962077 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0000587081d14ef1920a813cf3c7edaa" } new_table_name: "default.Table_ef823ba5464d429db5aa6b8c3ccc1934" modify_external_catalogs: true
I20250109 20:52:46.964951 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a455033b674e5a94fc37d0275eb313 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:46.967363 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 72a455033b674e5a94fc37d0275eb313 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a455033b674e5a94fc37d0275eb313 already in progress: creating tablet
I20250109 20:52:46.972060 1157 ts_tablet_manager.cc:1918] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.972800 1157 log.cc:1198] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/03e06e64448a4fb2bf41fc6e3d0361a6
I20250109 20:52:46.973400 1157 ts_tablet_manager.cc:1939] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:46.974977 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 03e06e64448a4fb2bf41fc6e3d0361a6 (table default.Table_609a0ae37cfd4b6d8939b4829beb84d6 [id=f06f85dff8684b1cac234aa2e35b6598]) successfully deleted
I20250109 20:52:46.982430 1157 tablet_replica.cc:331] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:46.982726 1157 raft_consensus.cc:2238] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:46.991150 1166 ts_tablet_manager.cc:1918] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:46.991293 1166 log.cc:1198] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/268d9969a46544d9954f71c77170e76c
I20250109 20:52:46.991641 1166 ts_tablet_manager.cc:1939] T 268d9969a46544d9954f71c77170e76c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:46.992363 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 248) took 1137 ms. Trace:
I20250109 20:52:46.992473 1166 rpcz_store.cc:276] 0109 20:52:45.855089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:45.855319 (+ 230us) service_pool.cc:225] Handling call
0109 20:52:46.794360 (+939041us) ts_tablet_manager.cc:1145] Deleting tablet 268d9969a46544d9954f71c77170e76c
0109 20:52:46.794378 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:46.949953 (+155575us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:46.991046 (+ 41093us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:46.992307 (+ 1261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":938310}
I20250109 20:52:46.993746 1176 tablet_bootstrap.cc:654] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:47.000569 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b31b9785ee21418ba13b426ba94cfa9b consensus state has no leader
I20250109 20:52:47.009052 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 268d9969a46544d9954f71c77170e76c (table default.Table_5285d784ef3043f397f8b3f5c0facc07 [id=42597c8e893d4f7084df9638e72d0306]) successfully deleted
I20250109 20:52:47.009240 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 247) took 1154 ms. Trace:
I20250109 20:52:47.009353 1166 rpcz_store.cc:276] 0109 20:52:45.854374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:45.854420 (+ 46us) service_pool.cc:225] Handling call
0109 20:52:47.009047 (+1154627us) ts_tablet_manager.cc:1145] Deleting tablet 607a78e281594e9a92c6aa63ee7bdf8d
0109 20:52:47.009059 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.009187 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151829}
W20250109 20:52:47.009925 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 607a78e281594e9a92c6aa63ee7bdf8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607a78e281594e9a92c6aa63ee7bdf8d already in progress: creating tablet
I20250109 20:52:47.010382 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 59e1e8754625463ea42a92ef65a89214 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.012301 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 254) took 1002 ms. Trace:
I20250109 20:52:47.012413 1166 rpcz_store.cc:276] 0109 20:52:46.009460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.009737 (+ 277us) service_pool.cc:225] Handling call
0109 20:52:47.012137 (+1002400us) ts_tablet_manager.cc:1145] Deleting tablet b221e048f6f6463e84e630386c304a05
0109 20:52:47.012148 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.012249 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1002122}
W20250109 20:52:47.014190 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b221e048f6f6463e84e630386c304a05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b221e048f6f6463e84e630386c304a05 already in progress: creating tablet
I20250109 20:52:47.021512 1157 raft_consensus.cc:2267] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.022383 887 tablet_service.cc:1467] Processing CreateTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 (DEFAULT_TABLE table=default.Table_c4e1ecf827064d4198b94eee6e6b4120 [id=87a978ea0af846f0b3ea56a3d1942d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.022927 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 246fd9ac96f54d008e8203e4580e7ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.023223 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 (DEFAULT_TABLE table=default.Table_c4e1ecf827064d4198b94eee6e6b4120 [id=87a978ea0af846f0b3ea56a3d1942d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.025971 886 tablet_service.cc:1467] Processing CreateTablet for tablet 165f6304acc849569a8e90d135a8e2d0 (DEFAULT_TABLE table=default.Table_10e1ff86f4e8433f8f5174dd6d3f19fb [id=4f170eeebe414cf8a63e96d92ac15320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.026003 895 tablet_service.cc:1467] Processing CreateTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 (DEFAULT_TABLE table=default.Table_6cca94c4ba974861b95038c43ebf833a [id=bf679a467a78433899ead891f63fed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.024544 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 (DEFAULT_TABLE table=default.Table_6cca94c4ba974861b95038c43ebf833a [id=bf679a467a78433899ead891f63fed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.026360 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165f6304acc849569a8e90d135a8e2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.026394 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8419d8cc7e475ea43dd263cae87aca (DEFAULT_TABLE table=default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.026682 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8419d8cc7e475ea43dd263cae87aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.026844 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686ed5f7d5524c54a8debe1bc610bd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.027774 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8419d8cc7e475ea43dd263cae87aca (DEFAULT_TABLE table=default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.027979 742 tablet_service.cc:1467] Processing CreateTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 (DEFAULT_TABLE table=default.Table_c4e1ecf827064d4198b94eee6e6b4120 [id=87a978ea0af846f0b3ea56a3d1942d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.028409 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 246fd9ac96f54d008e8203e4580e7ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.028381 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 165f6304acc849569a8e90d135a8e2d0 (DEFAULT_TABLE table=default.Table_10e1ff86f4e8433f8f5174dd6d3f19fb [id=4f170eeebe414cf8a63e96d92ac15320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.030086 744 tablet_service.cc:1467] Processing CreateTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 (DEFAULT_TABLE table=default.Table_6cca94c4ba974861b95038c43ebf833a [id=bf679a467a78433899ead891f63fed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.031780 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686ed5f7d5524c54a8debe1bc610bd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.033562 740 tablet_service.cc:1467] Processing CreateTablet for tablet 165f6304acc849569a8e90d135a8e2d0 (DEFAULT_TABLE table=default.Table_10e1ff86f4e8433f8f5174dd6d3f19fb [id=4f170eeebe414cf8a63e96d92ac15320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.033955 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165f6304acc849569a8e90d135a8e2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.035118 1157 ts_tablet_manager.cc:1905] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.035408 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8419d8cc7e475ea43dd263cae87aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.036094 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8419d8cc7e475ea43dd263cae87aca (DEFAULT_TABLE table=default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:47.041630 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a89a0b47d10414d9b6a9123af1f4ab3 consensus state has no leader
I20250109 20:52:47.042070 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3fa23057a60a4a3eb7755e799bc3b911
I20250109 20:52:47.042256 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8419d8cc7e475ea43dd263cae87aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.042907 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 59e1e8754625463ea42a92ef65a89214
I20250109 20:52:47.043370 449 catalog_manager.cc:6596] ReplaceTablet: tablet 59e1e8754625463ea42a92ef65a89214 deleted and replaced by tablet 85a95a31468b431fb4304cf6680016da
I20250109 20:52:47.043789 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3000b819a4cc4960909b98febd400631
I20250109 20:52:47.044188 452 catalog_manager.cc:6596] ReplaceTablet: tablet 3000b819a4cc4960909b98febd400631 deleted and replaced by tablet 1c65489a602141239c99e8e8dae22ab4
I20250109 20:52:47.045172 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e8754625463ea42a92ef65a89214 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a95a31468b431fb4304cf6680016da) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.045660 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.046341 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3000b819a4cc4960909b98febd400631 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c65489a602141239c99e8e8dae22ab4) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.046944 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3000b819a4cc4960909b98febd400631 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c65489a602141239c99e8e8dae22ab4) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.047364 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e8754625463ea42a92ef65a89214 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a95a31468b431fb4304cf6680016da) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.048676 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3000b819a4cc4960909b98febd400631 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c65489a602141239c99e8e8dae22ab4) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:47.049769 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3000b819a4cc4960909b98febd400631 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3000b819a4cc4960909b98febd400631 already in progress: creating tablet
I20250109 20:52:47.049913 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165f6304acc849569a8e90d135a8e2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.049973 1191 consensus_queue.cc:1035] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:47.054318 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e8754625463ea42a92ef65a89214 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a95a31468b431fb4304cf6680016da) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:47.055326 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 59e1e8754625463ea42a92ef65a89214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59e1e8754625463ea42a92ef65a89214 already in progress: creating tablet
I20250109 20:52:47.056624 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 246fd9ac96f54d008e8203e4580e7ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.060811 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.060967 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686ed5f7d5524c54a8debe1bc610bd25. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:47.062213 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3fa23057a60a4a3eb7755e799bc3b911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa23057a60a4a3eb7755e799bc3b911 already in progress: creating tablet
I20250109 20:52:47.063263 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.063510 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_65ca714c159440f4a1cc204e1c50d821" } new_table_name: "default.Table_3a269f56a3b446c1973d3b9a24f0ec8f" modify_external_catalogs: true
I20250109 20:52:47.065286 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_87926cb29c4d4a6483062503677f2798"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.065944 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87926cb29c4d4a6483062503677f2798 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.067631 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fdc4d53d46874cb0a25cd71dfa7a3960"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.068116 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdc4d53d46874cb0a25cd71dfa7a3960 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.078240 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1d52e3d34dc14e729a515dc24e0787d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.078503 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_245bc15e47a24b4f961bb77ad04a5443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.078814 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d52e3d34dc14e729a515dc24e0787d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:47.079113 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_245bc15e47a24b4f961bb77ad04a5443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.080137 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3349fa7d981e4c0cbaea5fce12ad7c67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.080570 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3349fa7d981e4c0cbaea5fce12ad7c67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.085156 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 258) took 1054 ms. Trace:
I20250109 20:52:47.089123 1166 rpcz_store.cc:276] 0109 20:52:46.031103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.039188 (+ 8085us) service_pool.cc:225] Handling call
0109 20:52:47.014118 (+974930us) ts_tablet_manager.cc:1145] Deleting tablet 91dbccae0af24a629600a92e7eadfee7
0109 20:52:47.014128 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.085094 (+ 70966us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":974668}
I20250109 20:52:47.089591 1158 tablet_bootstrap.cc:492] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.089730 1158 ts_tablet_manager.cc:1397] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.153s user 0.001s sys 0.007s
I20250109 20:52:47.089802 1165 tablet_bootstrap.cc:492] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.089953 1165 ts_tablet_manager.cc:1397] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.213s user 0.020s sys 0.039s
I20250109 20:52:47.090657 1158 raft_consensus.cc:357] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.090713 1165 raft_consensus.cc:357] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.090909 1165 raft_consensus.cc:383] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.090991 1165 raft_consensus.cc:738] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.091285 1158 raft_consensus.cc:383] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.091225 1165 consensus_queue.cc:260] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.091389 1158 raft_consensus.cc:738] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.093312 1158 consensus_queue.cc:260] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.093434 1165 ts_tablet_manager.cc:1428] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:47.093730 1165 tablet_bootstrap.cc:492] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.095326 1158 ts_tablet_manager.cc:1428] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:47.096060 1165 tablet_bootstrap.cc:654] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.096298 1158 tablet_bootstrap.cc:492] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.096894 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 257) took 1078 ms. Trace:
I20250109 20:52:47.098348 1158 tablet_bootstrap.cc:654] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:47.100991 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 91dbccae0af24a629600a92e7eadfee7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91dbccae0af24a629600a92e7eadfee7 already in progress: creating tablet
I20250109 20:52:47.103272 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.104615 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e06e64448a4fb2bf41fc6e3d0361a6 with delete_type TABLET_DATA_DELETED (replaced by tablet cba0ffb1b394431ba30918911cdee7b7) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.104902 1160 tablet_replica.cc:331] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.105113 1160 raft_consensus.cc:2238] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.105279 1160 raft_consensus.cc:2267] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.106168 1160 ts_tablet_manager.cc:1905] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.114902 1157 ts_tablet_manager.cc:1918] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.115054 1157 log.cc:1198] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/268d9969a46544d9954f71c77170e76c
W20250109 20:52:47.115077 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 110483us
I20250109 20:52:47.115451 1157 ts_tablet_manager.cc:1939] T 268d9969a46544d9954f71c77170e76c P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.112203 1165 tablet_bootstrap.cc:492] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.116230 1165 ts_tablet_manager.cc:1397] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250109 20:52:47.116580 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 268d9969a46544d9954f71c77170e76c (table default.Table_5285d784ef3043f397f8b3f5c0facc07 [id=42597c8e893d4f7084df9638e72d0306]) successfully deleted
W20250109 20:52:47.117302 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet fbde2eecdaf4470dbc74b0c59f31643c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbde2eecdaf4470dbc74b0c59f31643c already in progress: creating tablet
I20250109 20:52:47.118214 1165 raft_consensus.cc:357] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.118439 1165 raft_consensus.cc:383] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.118515 1165 raft_consensus.cc:738] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:52:47.119223 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 540531463c184b9abf226d7487939e60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 540531463c184b9abf226d7487939e60 already in progress: creating tablet
W20250109 20:52:47.119611 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 72a455033b674e5a94fc37d0275eb313 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a455033b674e5a94fc37d0275eb313 already in progress: creating tablet
I20250109 20:52:47.120149 1157 tablet_replica.cc:331] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:52:47.119920 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3000b819a4cc4960909b98febd400631 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3000b819a4cc4960909b98febd400631 already in progress: creating tablet
I20250109 20:52:47.120602 1157 raft_consensus.cc:2238] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:47.120615 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 59e1e8754625463ea42a92ef65a89214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59e1e8754625463ea42a92ef65a89214 already in progress: creating tablet
I20250109 20:52:47.120862 1157 raft_consensus.cc:2267] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:47.120873 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3fa23057a60a4a3eb7755e799bc3b911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa23057a60a4a3eb7755e799bc3b911 already in progress: creating tablet
I20250109 20:52:47.121454 1165 consensus_queue.cc:260] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.124614 1157 ts_tablet_manager.cc:1905] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.124966 1296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:52:47.126010 1296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:52:47.126295 1160 ts_tablet_manager.cc:1918] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.126312 1296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:52:47.126551 1160 log.cc:1198] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/03e06e64448a4fb2bf41fc6e3d0361a6
I20250109 20:52:47.127559 1160 ts_tablet_manager.cc:1939] T 03e06e64448a4fb2bf41fc6e3d0361a6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.128175 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 03e06e64448a4fb2bf41fc6e3d0361a6 (table default.Table_609a0ae37cfd4b6d8939b4829beb84d6 [id=f06f85dff8684b1cac234aa2e35b6598]) successfully deleted
I20250109 20:52:47.136492 1165 ts_tablet_manager.cc:1428] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.019s user 0.002s sys 0.003s
I20250109 20:52:47.112331 1166 rpcz_store.cc:276] 0109 20:52:46.018537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.018595 (+ 58us) service_pool.cc:225] Handling call
0109 20:52:47.096677 (+1078082us) ts_tablet_manager.cc:1145] Deleting tablet 66e4bfa188b545349f358adfd0bc714b
0109 20:52:47.096688 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.096841 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050082}
W20250109 20:52:47.138041 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 66e4bfa188b545349f358adfd0bc714b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66e4bfa188b545349f358adfd0bc714b already in progress: creating tablet
I20250109 20:52:47.137856 1165 tablet_bootstrap.cc:492] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.139750 1165 tablet_bootstrap.cc:654] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.142834 1176 tablet_bootstrap.cc:492] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:47.143026 1166 tablet_replica.cc:331] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:47.143234 1166 raft_consensus.cc:2238] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.143415 1166 raft_consensus.cc:2267] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.144043 1166 ts_tablet_manager.cc:1905] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.144789 1176 ts_tablet_manager.cc:1397] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.300s user 0.001s sys 0.006s
I20250109 20:52:47.145723 1410 consensus_queue.cc:1035] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:47.145763 1176 raft_consensus.cc:357] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.146061 1176 raft_consensus.cc:383] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.146134 1176 raft_consensus.cc:738] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.146368 1176 consensus_queue.cc:260] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.147635 1176 ts_tablet_manager.cc:1428] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:47.148039 1176 tablet_bootstrap.cc:492] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:47.165668 1157 ts_tablet_manager.cc:1918] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.165854 1157 log.cc:1198] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/32eb4dfcf96c4a05a55082e90ae04467
I20250109 20:52:47.167479 1165 tablet_bootstrap.cc:492] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.167637 1165 ts_tablet_manager.cc:1397] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20250109 20:52:47.169306 1165 raft_consensus.cc:357] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.169520 1165 raft_consensus.cc:383] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.169610 1165 raft_consensus.cc:738] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.171228 1165 consensus_queue.cc:260] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.173228 1157 ts_tablet_manager.cc:1939] T 32eb4dfcf96c4a05a55082e90ae04467 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.171784 1165 ts_tablet_manager.cc:1428] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:47.173832 1165 tablet_bootstrap.cc:492] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.175611 1165 tablet_bootstrap.cc:654] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.178361 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 32eb4dfcf96c4a05a55082e90ae04467 (table default.Table_384c816ede6b4c558524a02495bb3dc0 [id=756ad36069be45acaadb834d6ad41d05]) successfully deleted
I20250109 20:52:47.181330 1392 raft_consensus.cc:491] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.181488 1392 raft_consensus.cc:513] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.182369 1392 leader_election.cc:290] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:47.182996 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4866f6007d411da5cf4832fa92ece9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:47.183660 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49b7725a1f7647cd9b136c55bd8ed301" } modify_external_catalogs: true
I20250109 20:52:47.183717 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_171f660e57e54000a542c3e8bcd0bfb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.183866 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49b7725a1f7647cd9b136c55bd8ed301" } modify_external_catalogs: true
W20250109 20:52:47.184336 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_171f660e57e54000a542c3e8bcd0bfb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.184402 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f7702e93eb894ea5bbf0af322f94c29c" } modify_external_catalogs: true
I20250109 20:52:47.184574 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f7702e93eb894ea5bbf0af322f94c29c" } modify_external_catalogs: true
W20250109 20:52:47.187858 990 leader_election.cc:343] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:47.189049 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80da78917f0b4931ab1705371c1fe5dd has no consensus state
I20250109 20:52:47.190064 1158 tablet_bootstrap.cc:492] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.190215 1158 ts_tablet_manager.cc:1397] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.094s user 0.024s sys 0.012s
I20250109 20:52:47.190677 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2307b80fa70945568bfb4848975275b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.191241 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2307b80fa70945568bfb4848975275b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.191260 1158 raft_consensus.cc:357] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.191515 1158 raft_consensus.cc:383] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.191576 1158 raft_consensus.cc:738] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.191802 1158 consensus_queue.cc:260] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.192750 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0a104f68c5934305bc28de2aa457e751" } new_table_name: "default.Table_c6f02c8ee12943ba91552c8fa3eb5118" modify_external_catalogs: true
I20250109 20:52:47.194200 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_19363d9282fe42228ab784489a2b9a17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.194645 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19363d9282fe42228ab784489a2b9a17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.195802 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7d9ef737f1e74ae1b6f66968d5c0b586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.196202 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d9ef737f1e74ae1b6f66968d5c0b586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.198223 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d044013ec92a4c9186309687a275377e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.198624 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d044013ec92a4c9186309687a275377e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:47.199797 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd3d0bfb39db40758e82be511d033b4c consensus state has no leader
I20250109 20:52:47.200016 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bb480bfd701049cc8b814e0fe8b4c9ce" } modify_external_catalogs: true
I20250109 20:52:47.200237 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bb480bfd701049cc8b814e0fe8b4c9ce" } modify_external_catalogs: true
I20250109 20:52:47.204655 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 268d9969a46544d9954f71c77170e76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.204893 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.204993 1158 ts_tablet_manager.cc:1428] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250109 20:52:47.205415 1158 tablet_bootstrap.cc:492] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.208607 1158 tablet_bootstrap.cc:654] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:47.209571 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 32eb4dfcf96c4a05a55082e90ae04467 because the tablet was not found. No further retry: Not found: Tablet not found: 32eb4dfcf96c4a05a55082e90ae04467
I20250109 20:52:47.217701 1160 tablet_replica.cc:331] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.217955 1160 raft_consensus.cc:2238] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.219252 1160 raft_consensus.cc:2267] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.205611 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4866f6007d411da5cf4832fa92ece9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:47.220252 1160 ts_tablet_manager.cc:1905] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:47.220949 989 leader_election.cc:343] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:47.221127 989 leader_election.cc:304] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:47.221557 1392 raft_consensus.cc:2743] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:47.227469 1176 tablet_bootstrap.cc:654] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.229195 1160 ts_tablet_manager.cc:1918] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.233062 1160 log.cc:1198] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/268d9969a46544d9954f71c77170e76c
I20250109 20:52:47.233728 1160 ts_tablet_manager.cc:1939] T 268d9969a46544d9954f71c77170e76c P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.234936 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 268d9969a46544d9954f71c77170e76c (table default.Table_5285d784ef3043f397f8b3f5c0facc07 [id=42597c8e893d4f7084df9638e72d0306]) successfully deleted
I20250109 20:52:47.235360 1176 tablet_bootstrap.cc:492] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:47.235518 1176 ts_tablet_manager.cc:1397] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.088s user 0.001s sys 0.002s
I20250109 20:52:47.236241 1176 raft_consensus.cc:357] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.236466 1176 raft_consensus.cc:383] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.236550 1176 raft_consensus.cc:738] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:52:47.237183 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 65208.1us
I20250109 20:52:47.244943 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet c528a1a9ae0541ba9d4a80e38e743969 (table default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]). Expected version 1 got 0
W20250109 20:52:47.245576 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ed134863bf642e79c26e5bf75b248b0 consensus state has no leader
I20250109 20:52:47.246912 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_58060dd58dce4f7a8afa4d8c891c3ee3" } modify_external_catalogs: true
I20250109 20:52:47.247085 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_58060dd58dce4f7a8afa4d8c891c3ee3" } modify_external_catalogs: true
I20250109 20:52:47.247406 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7d999163ab094d83933d7da655cb380a" } new_table_name: "default.Table_dc8c9fd3f3844ff28cbe535e71493c94" modify_external_catalogs: true
I20250109 20:52:47.251041 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 (DEFAULT_TABLE table=default.Table_13f472ba51244613ab6c1c0cc5e9f9e4 [id=3fc24473de2845628a019824b932a4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.251252 887 tablet_service.cc:1467] Processing CreateTablet for tablet ea477dc845614ea78e71e45f862b57ff (DEFAULT_TABLE table=default.Table_1b9aa886edd542fc8039fd921d1048c8 [id=d64f7249495e433fa9c57277ee5eaba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.251516 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4160bb4ba40347bda48f8fa88f8b6001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.251761 886 tablet_service.cc:1467] Processing CreateTablet for tablet 8f420b7e623243e093d26c95276c5af9 (DEFAULT_TABLE table=default.Table_2e446345a181499d86750bfaca472d62 [id=ec17429d6df548118b1e53e77bac80c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.252072 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f420b7e623243e093d26c95276c5af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.254765 742 tablet_service.cc:1467] Processing CreateTablet for tablet ea477dc845614ea78e71e45f862b57ff (DEFAULT_TABLE table=default.Table_1b9aa886edd542fc8039fd921d1048c8 [id=d64f7249495e433fa9c57277ee5eaba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.255177 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea477dc845614ea78e71e45f862b57ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.257017 895 tablet_service.cc:1467] Processing CreateTablet for tablet 18f85944f66543fe8dd634ad0da0728d (DEFAULT_TABLE table=default.Table_9cbf7236e62e4c22a28e6ccb34f76495 [id=f3bf80bcfbb24031a4050f6f53569e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.257144 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea477dc845614ea78e71e45f862b57ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.257398 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f85944f66543fe8dd634ad0da0728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.260936 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 268d9969a46544d9954f71c77170e76c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.261379 899 tablet_service.cc:1467] Processing CreateTablet for tablet e090f72eb1e046319b825ee5fe126332 (DEFAULT_TABLE table=default.Table_e9297199b4064b5f889530ac2757c3ad [id=53929cac72e44474a2dbc7ae26524da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.261865 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e090f72eb1e046319b825ee5fe126332. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.263175 1032 tablet_service.cc:1467] Processing CreateTablet for tablet ea477dc845614ea78e71e45f862b57ff (DEFAULT_TABLE table=default.Table_1b9aa886edd542fc8039fd921d1048c8 [id=d64f7249495e433fa9c57277ee5eaba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.264882 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 32eb4dfcf96c4a05a55082e90ae04467 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.265239 897 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a9aba1c7b424db21c30b2e92323c9 (DEFAULT_TABLE table=default.Table_ce7eb75c66cc418e8f6b1729d7d51e57 [id=2457c6a6333f432da92aa163a82cebf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.265250 744 tablet_service.cc:1467] Processing CreateTablet for tablet 18f85944f66543fe8dd634ad0da0728d (DEFAULT_TABLE table=default.Table_9cbf7236e62e4c22a28e6ccb34f76495 [id=f3bf80bcfbb24031a4050f6f53569e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.265558 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6a9aba1c7b424db21c30b2e92323c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.265558 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f85944f66543fe8dd634ad0da0728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.266144 891 tablet_service.cc:1467] Processing CreateTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa (DEFAULT_TABLE table=default.Table_07f19c8f31bb48c598a7a2b8dc7187af [id=e0c7ea945a0a40e0a4a3e369c7455eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.266600 888 tablet_service.cc:1467] Processing CreateTablet for tablet 80da78917f0b4931ab1705371c1fe5dd (DEFAULT_TABLE table=default.Table_3a269f56a3b446c1973d3b9a24f0ec8f [id=16cb6983d685473c8d7aba4e39573aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.274230 1165 tablet_bootstrap.cc:492] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.274399 1165 ts_tablet_manager.cc:1397] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.007s
I20250109 20:52:47.275056 1160 tablet_replica.cc:331] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.275354 1160 raft_consensus.cc:2238] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:47.275698 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 268d9969a46544d9954f71c77170e76c because the tablet was not found. No further retry: Not found: Tablet not found: 268d9969a46544d9954f71c77170e76c
I20250109 20:52:47.276539 1160 raft_consensus.cc:2267] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.277472 1160 ts_tablet_manager.cc:1905] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.277590 449 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet c528a1a9ae0541ba9d4a80e38e743969 (table default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]). Expected version 1 got 0
I20250109 20:52:47.279042 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80da78917f0b4931ab1705371c1fe5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.280815 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68d23fb2df743f4afa2682f6d2e3efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.287784 447 master_service.cc:946] ReplaceTablet: received request to replace tablet ca816507d0c047668a51f6e7ebce18cb from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.289440 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_214cf06447eb4076abbed4a92f99fc79" } new_table_name: "default.Table_bd7a370b6cc2487da8595058ef89e0f5" modify_external_catalogs: true
I20250109 20:52:47.291064 1165 raft_consensus.cc:357] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.291380 1165 raft_consensus.cc:383] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.293598 1165 raft_consensus.cc:738] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.293993 1165 consensus_queue.cc:260] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.295565 1165 ts_tablet_manager.cc:1428] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250109 20:52:47.296619 1165 tablet_bootstrap.cc:492] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.298138 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_245bc15e47a24b4f961bb77ad04a5443" } new_table_name: "default.Table_711789d44eb242309002efa5bd06ffb1" modify_external_catalogs: true
I20250109 20:52:47.298420 1165 tablet_bootstrap.cc:654] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.299563 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.302428 742 tablet_service.cc:1467] Processing CreateTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 (DEFAULT_TABLE table=default.Table_13f472ba51244613ab6c1c0cc5e9f9e4 [id=3fc24473de2845628a019824b932a4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.302902 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4160bb4ba40347bda48f8fa88f8b6001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.304778 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 18f85944f66543fe8dd634ad0da0728d (DEFAULT_TABLE table=default.Table_9cbf7236e62e4c22a28e6ccb34f76495 [id=f3bf80bcfbb24031a4050f6f53569e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.305470 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 (DEFAULT_TABLE table=default.Table_13f472ba51244613ab6c1c0cc5e9f9e4 [id=3fc24473de2845628a019824b932a4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.310787 1160 ts_tablet_manager.cc:1918] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.311057 1160 log.cc:1198] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/32eb4dfcf96c4a05a55082e90ae04467
I20250109 20:52:47.311555 1160 ts_tablet_manager.cc:1939] T 32eb4dfcf96c4a05a55082e90ae04467 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.312103 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8f420b7e623243e093d26c95276c5af9 (DEFAULT_TABLE table=default.Table_2e446345a181499d86750bfaca472d62 [id=ec17429d6df548118b1e53e77bac80c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.312319 1165 tablet_bootstrap.cc:492] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.312502 1165 ts_tablet_manager.cc:1397] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.008s
I20250109 20:52:47.312547 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f420b7e623243e093d26c95276c5af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.312877 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 32eb4dfcf96c4a05a55082e90ae04467 (table default.Table_384c816ede6b4c558524a02495bb3dc0 [id=756ad36069be45acaadb834d6ad41d05]) successfully deleted
I20250109 20:52:47.313269 1165 raft_consensus.cc:357] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.313575 1165 raft_consensus.cc:383] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.313701 1165 raft_consensus.cc:738] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.314005 1165 consensus_queue.cc:260] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.314352 1160 tablet_replica.cc:331] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.315718 1160 raft_consensus.cc:2238] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.316251 1160 raft_consensus.cc:2267] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.314880 1165 ts_tablet_manager.cc:1428] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:47.317191 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.317238 1165 tablet_bootstrap.cc:492] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.317461 1157 tablet_replica.cc:331] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.317631 1157 raft_consensus.cc:2238] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.317811 1157 raft_consensus.cc:2267] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.317873 740 tablet_service.cc:1467] Processing CreateTablet for tablet e090f72eb1e046319b825ee5fe126332 (DEFAULT_TABLE table=default.Table_e9297199b4064b5f889530ac2757c3ad [id=53929cac72e44474a2dbc7ae26524da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.318289 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e090f72eb1e046319b825ee5fe126332. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.318789 1165 tablet_bootstrap.cc:654] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.318768 1157 ts_tablet_manager.cc:1905] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.323279 1160 ts_tablet_manager.cc:1905] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.324498 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 8f420b7e623243e093d26c95276c5af9 (DEFAULT_TABLE table=default.Table_2e446345a181499d86750bfaca472d62 [id=ec17429d6df548118b1e53e77bac80c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.325404 1035 tablet_service.cc:1467] Processing CreateTablet for tablet e090f72eb1e046319b825ee5fe126332 (DEFAULT_TABLE table=default.Table_e9297199b4064b5f889530ac2757c3ad [id=53929cac72e44474a2dbc7ae26524da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.326161 740 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a9aba1c7b424db21c30b2e92323c9 (DEFAULT_TABLE table=default.Table_ce7eb75c66cc418e8f6b1729d7d51e57 [id=2457c6a6333f432da92aa163a82cebf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.326306 1157 ts_tablet_manager.cc:1918] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.326457 1157 log.cc:1198] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/825efab5ff7b492ca3c7e3fe11ece098
I20250109 20:52:47.326807 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6a9aba1c7b424db21c30b2e92323c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.326903 1157 ts_tablet_manager.cc:1939] T 825efab5ff7b492ca3c7e3fe11ece098 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.330471 1160 ts_tablet_manager.cc:1918] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.330636 1160 log.cc:1198] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/66c6016d88b54eb390f880945ddb1ea6
I20250109 20:52:47.331012 1160 ts_tablet_manager.cc:1939] T 66c6016d88b54eb390f880945ddb1ea6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.331928 1165 tablet_bootstrap.cc:492] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.331938 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 66c6016d88b54eb390f880945ddb1ea6 (table default.Table_f9f28decdfb64455809f7b283552a4a5 [id=ae7fb9326c2c49a593d1111d45ea424f]) successfully deleted
I20250109 20:52:47.332118 1165 ts_tablet_manager.cc:1397] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250109 20:52:47.333179 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6a9aba1c7b424db21c30b2e92323c9 (DEFAULT_TABLE table=default.Table_ce7eb75c66cc418e8f6b1729d7d51e57 [id=2457c6a6333f432da92aa163a82cebf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.333117 1165 raft_consensus.cc:357] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.333351 1165 raft_consensus.cc:383] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.333431 1165 raft_consensus.cc:738] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.333562 753 tablet_service.cc:1467] Processing CreateTablet for tablet 80da78917f0b4931ab1705371c1fe5dd (DEFAULT_TABLE table=default.Table_3a269f56a3b446c1973d3b9a24f0ec8f [id=16cb6983d685473c8d7aba4e39573aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.333667 1165 consensus_queue.cc:260] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.333920 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80da78917f0b4931ab1705371c1fe5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.334021 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 825efab5ff7b492ca3c7e3fe11ece098 (table default.Table_7a97ba56daf04341bc72ea1ffe462b8c [id=9c14de13a61b45f78bacd804507a2b14]) successfully deleted
I20250109 20:52:47.334403 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 80da78917f0b4931ab1705371c1fe5dd (DEFAULT_TABLE table=default.Table_3a269f56a3b446c1973d3b9a24f0ec8f [id=16cb6983d685473c8d7aba4e39573aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.334609 1165 ts_tablet_manager.cc:1428] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250109 20:52:47.335052 1020 tablet_service.cc:1467] Processing CreateTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa (DEFAULT_TABLE table=default.Table_07f19c8f31bb48c598a7a2b8dc7187af [id=e0c7ea945a0a40e0a4a3e369c7455eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.337218 740 tablet_service.cc:1467] Processing CreateTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa (DEFAULT_TABLE table=default.Table_07f19c8f31bb48c598a7a2b8dc7187af [id=e0c7ea945a0a40e0a4a3e369c7455eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.337597 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68d23fb2df743f4afa2682f6d2e3efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.236763 1176 consensus_queue.cc:260] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.335625 1165 tablet_bootstrap.cc:492] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.341948 1176 ts_tablet_manager.cc:1428] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.106s user 0.002s sys 0.000s
I20250109 20:52:47.342334 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6a9aba1c7b424db21c30b2e92323c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.342586 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e090f72eb1e046319b825ee5fe126332. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.342845 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80da78917f0b4931ab1705371c1fe5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.343297 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68d23fb2df743f4afa2682f6d2e3efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.343971 1165 tablet_bootstrap.cc:654] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.346735 1158 tablet_bootstrap.cc:492] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.346887 1158 ts_tablet_manager.cc:1397] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.142s user 0.015s sys 0.031s
I20250109 20:52:47.347764 1158 raft_consensus.cc:357] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.348011 1158 raft_consensus.cc:383] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.348100 1158 raft_consensus.cc:738] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.348380 1158 consensus_queue.cc:260] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.350082 1158 ts_tablet_manager.cc:1428] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:47.350565 1158 tablet_bootstrap.cc:492] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.351763 1165 tablet_bootstrap.cc:492] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.351928 1165 ts_tablet_manager.cc:1397] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20250109 20:52:47.352800 1165 raft_consensus.cc:357] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.353052 1165 raft_consensus.cc:383] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.353138 1165 raft_consensus.cc:738] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.353420 1165 consensus_queue.cc:260] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.355171 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f85944f66543fe8dd634ad0da0728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.357602 1176 tablet_bootstrap.cc:492] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:47.359153 1176 tablet_bootstrap.cc:654] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.359380 1165 ts_tablet_manager.cc:1428] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250109 20:52:47.361824 1165 tablet_bootstrap.cc:492] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.363466 1158 tablet_bootstrap.cc:654] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.366654 1165 tablet_bootstrap.cc:654] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.371520 1166 ts_tablet_manager.cc:1918] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.371652 1166 log.cc:1198] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/1044c212aada4df382d147c9a15ea4a7
I20250109 20:52:47.371886 1165 tablet_bootstrap.cc:492] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.372030 1166 ts_tablet_manager.cc:1939] T 1044c212aada4df382d147c9a15ea4a7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:47.372181 1165 ts_tablet_manager.cc:1397] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20250109 20:52:47.372821 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 250) took 1411 ms. Trace:
I20250109 20:52:47.372952 1166 rpcz_store.cc:276] 0109 20:52:45.961683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:45.965192 (+ 3509us) service_pool.cc:225] Handling call
0109 20:52:47.138161 (+1172969us) ts_tablet_manager.cc:1145] Deleting tablet 1044c212aada4df382d147c9a15ea4a7
0109 20:52:47.138173 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.229864 (+ 91691us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:47.359502 (+129638us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:47.372773 (+ 13271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064314}
I20250109 20:52:47.373401 1165 raft_consensus.cc:357] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.373618 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 261) took 1230 ms. Trace:
I20250109 20:52:47.373739 1166 rpcz_store.cc:276] 0109 20:52:46.143257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.143309 (+ 52us) service_pool.cc:225] Handling call
0109 20:52:47.373442 (+1230133us) ts_tablet_manager.cc:1145] Deleting tablet a58d439254f8418091e3f23c8c274240
0109 20:52:47.373451 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.373569 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229905}
I20250109 20:52:47.373828 1165 raft_consensus.cc:383] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.374082 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f420b7e623243e093d26c95276c5af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.374074 1165 raft_consensus.cc:738] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.374411 1165 consensus_queue.cc:260] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.376389 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4160bb4ba40347bda48f8fa88f8b6001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.378366 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea477dc845614ea78e71e45f862b57ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.379153 1165 ts_tablet_manager.cc:1428] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:52:47.382424 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 1044c212aada4df382d147c9a15ea4a7 (table default.Table_807cf97a78f84ba794b3d29b6e210081 [id=0eac4d9a91be4e02a4e766f780e4ca93]) successfully deleted
W20250109 20:52:47.382692 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a58d439254f8418091e3f23c8c274240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a58d439254f8418091e3f23c8c274240 already in progress: creating tablet
I20250109 20:52:47.387466 1165 tablet_bootstrap.cc:492] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.388387 1158 tablet_bootstrap.cc:492] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.388585 1158 ts_tablet_manager.cc:1397] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20250109 20:52:47.390007 1158 raft_consensus.cc:357] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.390233 1158 raft_consensus.cc:383] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.390311 1158 raft_consensus.cc:738] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.390547 1158 consensus_queue.cc:260] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.391858 1158 ts_tablet_manager.cc:1428] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250109 20:52:47.392469 1158 tablet_bootstrap.cc:492] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.394254 1165 tablet_bootstrap.cc:654] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.394253 1158 tablet_bootstrap.cc:654] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.396996 1165 tablet_bootstrap.cc:492] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.397287 1165 ts_tablet_manager.cc:1397] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20250109 20:52:47.397433 1158 tablet_bootstrap.cc:492] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.397598 1158 ts_tablet_manager.cc:1397] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250109 20:52:47.398450 1158 raft_consensus.cc:357] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.398589 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.398648 1158 raft_consensus.cc:383] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.398238 1165 raft_consensus.cc:357] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.399046 1158 raft_consensus.cc:738] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:47.399300 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 66c6016d88b54eb390f880945ddb1ea6 because the tablet was not found. No further retry: Not found: Tablet not found: 66c6016d88b54eb390f880945ddb1ea6
I20250109 20:52:47.399338 1158 consensus_queue.cc:260] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.399562 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_07f19c8f31bb48c598a7a2b8dc7187af" } modify_external_catalogs: true
I20250109 20:52:47.399745 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_07f19c8f31bb48c598a7a2b8dc7187af" } modify_external_catalogs: true
I20250109 20:52:47.399937 1165 raft_consensus.cc:383] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.400094 1165 raft_consensus.cc:738] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.400396 1165 consensus_queue.cc:260] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.403533 1158 ts_tablet_manager.cc:1428] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:52:47.405285 1158 tablet_bootstrap.cc:492] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.406286 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6016d88b54eb390f880945ddb1ea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.406544 1157 tablet_replica.cc:331] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.406685 1165 ts_tablet_manager.cc:1428] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:52:47.406708 1157 raft_consensus.cc:2238] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.406841 1157 raft_consensus.cc:2267] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.407114 1158 tablet_bootstrap.cc:654] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.407768 1157 ts_tablet_manager.cc:1905] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.415122 1158 tablet_bootstrap.cc:492] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.415349 1158 ts_tablet_manager.cc:1397] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20250109 20:52:47.416146 1158 raft_consensus.cc:357] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.416429 1158 raft_consensus.cc:383] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.416539 1158 raft_consensus.cc:738] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.416815 1158 consensus_queue.cc:260] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.417654 1157 ts_tablet_manager.cc:1918] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.420167 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 825efab5ff7b492ca3c7e3fe11ece098 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.420578 1160 tablet_replica.cc:331] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.420814 1160 raft_consensus.cc:2238] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.420979 1160 raft_consensus.cc:2267] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.422025 1160 ts_tablet_manager.cc:1905] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.422200 899 tablet_service.cc:1514] Processing DeleteTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.422462 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044c212aada4df382d147c9a15ea4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.422973 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.423529 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 59e1e8754625463ea42a92ef65a89214 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.424302 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 72a455033b674e5a94fc37d0275eb313 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.425078 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acadf60d566407aabc1a2395d1d235f with delete_type TABLET_DATA_DELETED (replaced by tablet f672cbd6dffc49dbb69992ae7db979ea) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.425575 1165 tablet_bootstrap.cc:492] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.426962 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 1925a82eb711409ab2c5429c61edf9d8 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.427734 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_49bb1f7179c6420c83e530237052e180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.428246 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49bb1f7179c6420c83e530237052e180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.428839 1160 ts_tablet_manager.cc:1918] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.429068 1160 log.cc:1198] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/825efab5ff7b492ca3c7e3fe11ece098
I20250109 20:52:47.429481 1160 ts_tablet_manager.cc:1939] T 825efab5ff7b492ca3c7e3fe11ece098 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.429901 1165 tablet_bootstrap.cc:654] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.430079 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 825efab5ff7b492ca3c7e3fe11ece098 (table default.Table_7a97ba56daf04341bc72ea1ffe462b8c [id=9c14de13a61b45f78bacd804507a2b14]) successfully deleted
I20250109 20:52:47.430188 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 1044c212aada4df382d147c9a15ea4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.430346 1157 log.cc:1198] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/66c6016d88b54eb390f880945ddb1ea6
I20250109 20:52:47.430799 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f865acd2c9aa47ec8ebc3bf90d81c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.424408 1268 raft_consensus.cc:491] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.431524 1268 raft_consensus.cc:513] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.432197 1268 leader_election.cc:290] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:47.429777 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acadf60d566407aabc1a2395d1d235f with delete_type TABLET_DATA_DELETED (replaced by tablet f672cbd6dffc49dbb69992ae7db979ea) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.433598 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acadf60d566407aabc1a2395d1d235f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:47.434005 1054 raft_consensus.cc:2463] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:47.430841 1157 ts_tablet_manager.cc:1939] T 66c6016d88b54eb390f880945ddb1ea6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.434535 713 leader_election.cc:304] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:47.434885 1268 raft_consensus.cc:2798] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:47.434990 1268 raft_consensus.cc:491] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:47.435076 1268 raft_consensus.cc:3054] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.435282 1157 tablet_replica.cc:331] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.435439 1157 raft_consensus.cc:2238] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.435585 1157 raft_consensus.cc:2267] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.436381 1157 ts_tablet_manager.cc:1905] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.430387 1160 tablet_replica.cc:331] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.433115 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acadf60d566407aabc1a2395d1d235f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:47.436651 1160 raft_consensus.cc:2238] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.436648 918 raft_consensus.cc:2463] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:47.436873 1160 raft_consensus.cc:2267] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.437827 1160 ts_tablet_manager.cc:1905] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.437722 1268 raft_consensus.cc:513] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.439177 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acadf60d566407aabc1a2395d1d235f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:47.439420 918 raft_consensus.cc:3054] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.442605 918 raft_consensus.cc:2463] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:47.443147 712 leader_election.cc:304] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2; no voters:
I20250109 20:52:47.418895 1158 ts_tablet_manager.cc:1428] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 20:52:47.444126 1160 ts_tablet_manager.cc:1918] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.444257 1160 log.cc:1198] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f865acd2c9aa47ec8ebc3bf90d81c21e
I20250109 20:52:47.444622 1160 ts_tablet_manager.cc:1939] T f865acd2c9aa47ec8ebc3bf90d81c21e P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.445132 1158 tablet_bootstrap.cc:492] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.446681 1268 leader_election.cc:290] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:47.447048 1268 raft_consensus.cc:2798] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:47.447404 1268 raft_consensus.cc:695] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:47.447939 1268 consensus_queue.cc:237] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.447947 1158 tablet_bootstrap.cc:654] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.452126 1165 tablet_bootstrap.cc:492] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.452380 1165 ts_tablet_manager.cc:1397] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.012s
I20250109 20:52:47.454008 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 66c6016d88b54eb390f880945ddb1ea6 (table default.Table_f9f28decdfb64455809f7b283552a4a5 [id=ae7fb9326c2c49a593d1111d45ea424f]) successfully deleted
I20250109 20:52:47.446398 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acadf60d566407aabc1a2395d1d235f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:47.454735 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 264) took 1235 ms. Trace:
I20250109 20:52:47.454905 1054 raft_consensus.cc:3054] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.454995 1158 tablet_bootstrap.cc:492] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.455134 1158 ts_tablet_manager.cc:1397] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20250109 20:52:47.455163 1165 raft_consensus.cc:357] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.455441 1165 raft_consensus.cc:383] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.455781 1165 raft_consensus.cc:738] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.456097 1165 consensus_queue.cc:260] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.457139 1054 raft_consensus.cc:2463] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:47.458437 1165 ts_tablet_manager.cc:1428] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250109 20:52:47.459285 1165 tablet_bootstrap.cc:492] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.460906 1165 tablet_bootstrap.cc:654] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.456362 1158 raft_consensus.cc:357] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.461522 1158 raft_consensus.cc:383] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.461620 1158 raft_consensus.cc:738] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.461855 1158 consensus_queue.cc:260] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.462515 1158 ts_tablet_manager.cc:1428] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250109 20:52:47.462785 1158 tablet_bootstrap.cc:492] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.463438 1166 rpcz_store.cc:276] 0109 20:52:46.170873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.170961 (+ 88us) service_pool.cc:225] Handling call
0109 20:52:47.382906 (+1211945us) ts_tablet_manager.cc:1145] Deleting tablet 95735385d5ab4fe0b29e33bad14eb95c
0109 20:52:47.382915 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.405870 (+ 22955us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205197}
I20250109 20:52:47.464284 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 265) took 1286 ms. Trace:
I20250109 20:52:47.464421 1166 rpcz_store.cc:276] 0109 20:52:46.177325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.177420 (+ 95us) service_pool.cc:225] Handling call
0109 20:52:47.463902 (+1286482us) ts_tablet_manager.cc:1145] Deleting tablet cba68160be6145ddbbd2019c9b5f0076
0109 20:52:47.463915 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.464246 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274831}
I20250109 20:52:47.464974 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 267) took 1275 ms. Trace:
I20250109 20:52:47.465060 1166 rpcz_store.cc:276] 0109 20:52:46.189192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.190184 (+ 992us) service_pool.cc:225] Handling call
0109 20:52:47.464832 (+1274648us) ts_tablet_manager.cc:1145] Deleting tablet 7d7539cf9ece4befaae6c521d471ea57
0109 20:52:47.464843 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.464944 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240805}
I20250109 20:52:47.465281 1158 tablet_bootstrap.cc:654] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.465375 1166 tablet_replica.cc:331] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:47.465579 1166 raft_consensus.cc:2238] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.465761 1166 raft_consensus.cc:2267] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.470958 1160 tablet_replica.cc:331] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.471086 1158 tablet_bootstrap.cc:492] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.471266 1158 ts_tablet_manager.cc:1397] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20250109 20:52:47.472076 1158 raft_consensus.cc:357] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.472301 1158 raft_consensus.cc:383] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.472390 1158 raft_consensus.cc:738] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:47.472498 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 95735385d5ab4fe0b29e33bad14eb95c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95735385d5ab4fe0b29e33bad14eb95c already in progress: creating tablet
W20250109 20:52:47.472743 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cba68160be6145ddbbd2019c9b5f0076 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cba68160be6145ddbbd2019c9b5f0076 already in progress: creating tablet
I20250109 20:52:47.472631 1158 consensus_queue.cc:260] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
W20250109 20:52:47.472905 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7d7539cf9ece4befaae6c521d471ea57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d7539cf9ece4befaae6c521d471ea57 already in progress: creating tablet
I20250109 20:52:47.473112 1157 ts_tablet_manager.cc:1918] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.473261 1160 raft_consensus.cc:2238] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.473390 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f865acd2c9aa47ec8ebc3bf90d81c21e (table default.Table_82b8503333e64f94829491f22a40b92b [id=87109c5773c14f159f596b9d30844948]) successfully deleted
I20250109 20:52:47.473528 1160 raft_consensus.cc:2267] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.473455 1158 ts_tablet_manager.cc:1428] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:47.477589 1158 tablet_bootstrap.cc:492] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.478786 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 46d4cad7594346dfafdee8e4c49d74e6
I20250109 20:52:47.478876 1160 ts_tablet_manager.cc:1905] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.480086 1158 tablet_bootstrap.cc:654] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.480382 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.482873 1157 log.cc:1198] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/1044c212aada4df382d147c9a15ea4a7
I20250109 20:52:47.483700 1165 tablet_bootstrap.cc:492] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.483830 1165 ts_tablet_manager.cc:1397] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.003s
I20250109 20:52:47.483840 1157 ts_tablet_manager.cc:1939] T 1044c212aada4df382d147c9a15ea4a7 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.484555 1165 raft_consensus.cc:357] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.484838 1165 raft_consensus.cc:383] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.484933 1165 raft_consensus.cc:738] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.485316 1165 consensus_queue.cc:260] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.486035 446 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet ed4866f6007d411da5cf4832fa92ece9 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]). Expected version 2 got 0
I20250109 20:52:47.486768 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_69f93e68f71f4bc589abb0edbc340e6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.487259 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69f93e68f71f4bc589abb0edbc340e6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.488183 1165 ts_tablet_manager.cc:1428] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:47.489717 1157 tablet_replica.cc:331] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.490160 1165 tablet_bootstrap.cc:492] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.492635 1165 tablet_bootstrap.cc:654] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.493798 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 73e82535780c47ca8b9cd94b013b4b42
I20250109 20:52:47.495186 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.497512 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.498050 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.498540 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 1044c212aada4df382d147c9a15ea4a7 (table default.Table_807cf97a78f84ba794b3d29b6e210081 [id=0eac4d9a91be4e02a4e766f780e4ca93]) successfully deleted
I20250109 20:52:47.499183 1160 ts_tablet_manager.cc:1918] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.499370 1160 log.cc:1198] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/1044c212aada4df382d147c9a15ea4a7
I20250109 20:52:47.499742 1160 ts_tablet_manager.cc:1939] T 1044c212aada4df382d147c9a15ea4a7 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.499780 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.500023 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet f1589b79a56148b6ab2946943de39fb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b31bd4072941d5aaf8c43dada3ccc7) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.500993 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 1044c212aada4df382d147c9a15ea4a7 (table default.Table_807cf97a78f84ba794b3d29b6e210081 [id=0eac4d9a91be4e02a4e766f780e4ca93]) successfully deleted
I20250109 20:52:47.501466 1054 raft_consensus.cc:1270] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:47.502133 1160 tablet_replica.cc:331] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.502103 1436 consensus_queue.cc:1035] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:47.502612 1160 raft_consensus.cc:2238] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.503044 1160 raft_consensus.cc:2267] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.505116 918 raft_consensus.cc:1270] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:47.507310 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet ed4866f6007d411da5cf4832fa92ece9 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]). Expected version 2 got 0
I20250109 20:52:47.508455 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.508399 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b1cfdfcec71b4de6b182f4598f9e2601"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.508584 1448 consensus_queue.cc:1035] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:52:47.509019 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1cfdfcec71b4de6b182f4598f9e2601 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.509413 1166 ts_tablet_manager.cc:1905] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.511545 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.512200 1392 raft_consensus.cc:491] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.512351 1392 raft_consensus.cc:513] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.513065 1392 leader_election.cc:290] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:47.513715 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b47ee787ca4dc5a4a56ae582d15b8b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:47.514074 776 raft_consensus.cc:2463] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:47.514432 1157 raft_consensus.cc:2238] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:47.514947 1157 pending_rounds.cc:62] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Trying to abort 1 pending ops.
I20250109 20:52:47.515038 1157 pending_rounds.cc:69] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523642675535872 op_type: NO_OP noop_request { }
I20250109 20:52:47.515168 1157 raft_consensus.cc:2883] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:47.515249 990 leader_election.cc:304] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:47.516597 1157 raft_consensus.cc:2267] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.517706 1176 tablet_bootstrap.cc:492] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:47.517861 1176 ts_tablet_manager.cc:1397] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.004s
I20250109 20:52:47.518285 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b47ee787ca4dc5a4a56ae582d15b8b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:47.518512 919 raft_consensus.cc:2463] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:47.519310 1392 raft_consensus.cc:2798] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:47.519428 1392 raft_consensus.cc:491] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:47.519516 1392 raft_consensus.cc:3054] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.519997 1176 raft_consensus.cc:357] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.520300 1176 raft_consensus.cc:383] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.520393 1176 raft_consensus.cc:738] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.520700 1176 consensus_queue.cc:260] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.522377 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6716e7428ce240fca7ca5b02f564a5df
I20250109 20:52:47.522511 1157 ts_tablet_manager.cc:1905] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.525374 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6716e7428ce240fca7ca5b02f564a5df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.526399 1158 tablet_bootstrap.cc:492] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.526559 1158 ts_tablet_manager.cc:1397] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.011s
I20250109 20:52:47.526947 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170]). Expected version 2 got 0
I20250109 20:52:47.527652 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7dcbaf73ff544c3e8e804e28f8e20a6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.527676 1158 raft_consensus.cc:357] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.527974 1158 raft_consensus.cc:383] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.528095 1158 raft_consensus.cc:738] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:47.528425 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dcbaf73ff544c3e8e804e28f8e20a6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.528376 1158 consensus_queue.cc:260] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.529140 1158 ts_tablet_manager.cc:1428] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20250109 20:52:47.529551 1158 tablet_bootstrap.cc:492] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.530478 1157 ts_tablet_manager.cc:1918] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:47.530740 1157 log.cc:1198] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6acadf60d566407aabc1a2395d1d235f
I20250109 20:52:47.531196 1157 ts_tablet_manager.cc:1939] T 6acadf60d566407aabc1a2395d1d235f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.533437 1392 raft_consensus.cc:513] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.535547 1166 ts_tablet_manager.cc:1918] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.535714 1166 log.cc:1198] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f865acd2c9aa47ec8ebc3bf90d81c21e
I20250109 20:52:47.536166 1166 ts_tablet_manager.cc:1939] T f865acd2c9aa47ec8ebc3bf90d81c21e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:47.537591 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 6716e7428ce240fca7ca5b02f564a5df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.543272 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 272) took 1319 ms. Trace:
I20250109 20:52:47.543408 1166 rpcz_store.cc:276] 0109 20:52:46.223732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.223789 (+ 57us) service_pool.cc:225] Handling call
0109 20:52:47.465323 (+1241534us) ts_tablet_manager.cc:1145] Deleting tablet f865acd2c9aa47ec8ebc3bf90d81c21e
0109 20:52:47.465330 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.524631 (+ 59301us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:47.535408 (+ 10777us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:47.543199 (+ 7791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241142}
I20250109 20:52:47.544051 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 275) took 1305 ms. Trace:
I20250109 20:52:47.544159 1166 rpcz_store.cc:276] 0109 20:52:46.238853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.255823 (+ 16970us) service_pool.cc:225] Handling call
0109 20:52:47.543803 (+1287980us) ts_tablet_manager.cc:1145] Deleting tablet 8a5a6dbb823e4bd3afeb157b3c5771ba
0109 20:52:47.543819 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.543982 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244927}
I20250109 20:52:47.544670 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 288) took 1183 ms. Trace:
I20250109 20:52:47.544828 1166 rpcz_store.cc:276] 0109 20:52:46.361548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.361832 (+ 284us) service_pool.cc:225] Handling call
0109 20:52:47.544499 (+1182667us) ts_tablet_manager.cc:1145] Deleting tablet aa7d7695ff5946abb6ee576431d2e87f
0109 20:52:47.544511 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.544610 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171962}
I20250109 20:52:47.544881 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f865acd2c9aa47ec8ebc3bf90d81c21e (table default.Table_82b8503333e64f94829491f22a40b92b [id=87109c5773c14f159f596b9d30844948]) successfully deleted
I20250109 20:52:47.545269 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 289) took 1152 ms. Trace:
I20250109 20:52:47.545377 1166 rpcz_store.cc:276] 0109 20:52:46.393229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.393727 (+ 498us) service_pool.cc:225] Handling call
0109 20:52:47.545100 (+1151373us) ts_tablet_manager.cc:1145] Deleting tablet 41de7acfa7f941caa94bdbb888cf841e
0109 20:52:47.545111 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.545205 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151145}
W20250109 20:52:47.545516 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a5a6dbb823e4bd3afeb157b3c5771ba already in progress: creating tablet
I20250109 20:52:47.545841 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 291) took 1126 ms. Trace:
W20250109 20:52:47.545982 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet aa7d7695ff5946abb6ee576431d2e87f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa7d7695ff5946abb6ee576431d2e87f already in progress: creating tablet
I20250109 20:52:47.545953 1166 rpcz_store.cc:276] 0109 20:52:46.419217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.419632 (+ 415us) service_pool.cc:225] Handling call
0109 20:52:47.545670 (+1126038us) ts_tablet_manager.cc:1145] Deleting tablet 6acca65ecee94f8cbe3f2b89e744aa6c
0109 20:52:47.545680 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.545783 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1125754}
I20250109 20:52:47.546362 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 290) took 1130 ms. Trace:
W20250109 20:52:47.546386 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 41de7acfa7f941caa94bdbb888cf841e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41de7acfa7f941caa94bdbb888cf841e already in progress: creating tablet
I20250109 20:52:47.546465 1166 rpcz_store.cc:276] 0109 20:52:46.419011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.436290 (+ 17279us) service_pool.cc:225] Handling call
0109 20:52:47.546199 (+1109909us) ts_tablet_manager.cc:1145] Deleting tablet 2d96a569e6be4e01a26918d115cfbcf9
0109 20:52:47.546210 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.546296 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109366}
I20250109 20:52:47.546921 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 293) took 1102 ms. Trace:
I20250109 20:52:47.547127 1392 leader_election.cc:290] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:47.549978 1392 raft_consensus.cc:491] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.550287 1392 raft_consensus.cc:513] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.551393 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ecb55a61d2e94cbb88d2f44fff1aa8d9
I20250109 20:52:47.552274 1392 leader_election.cc:290] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:47.552517 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca816507d0c047668a51f6e7ebce18cb consensus state has no leader
I20250109 20:52:47.553177 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ca816507d0c047668a51f6e7ebce18cb
I20250109 20:52:47.553568 447 catalog_manager.cc:6596] ReplaceTablet: tablet ca816507d0c047668a51f6e7ebce18cb deleted and replaced by tablet 5ce0a941f0c14501a947a5c7e2a844e8
I20250109 20:52:47.553872 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b47ee787ca4dc5a4a56ae582d15b8b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:47.554100 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c65489a602141239c99e8e8dae22ab4 has no consensus state
I20250109 20:52:47.554164 919 raft_consensus.cc:3054] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.554301 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 8a5a6dbb823e4bd3afeb157b3c5771ba from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.554602 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1589b79a56148b6ab2946943de39fb2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:47.555073 989 leader_election.cc:343] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:47.555492 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 225046d45c1c43b488babf5da8de75d7 has no consensus state
I20250109 20:52:47.556553 895 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.556699 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ca816507d0c047668a51f6e7ebce18cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce0a941f0c14501a947a5c7e2a844e8) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.557706 919 raft_consensus.cc:2463] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:47.547037 1166 rpcz_store.cc:276] 0109 20:52:46.444063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.444552 (+ 489us) service_pool.cc:225] Handling call
0109 20:52:47.546751 (+1102199us) ts_tablet_manager.cc:1145] Deleting tablet 49c6f1833ff1453f90a6ec6435248a02
0109 20:52:47.546762 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.546869 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101707}
I20250109 20:52:47.560962 1166 tablet_replica.cc:331] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:47.561059 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b68d23fb2df743f4afa2682f6d2e3efa
I20250109 20:52:47.561997 1166 raft_consensus.cc:2238] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.562019 989 leader_election.cc:304] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:47.562175 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.562263 1166 raft_consensus.cc:2267] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.562337 1392 raft_consensus.cc:2798] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:47.562486 1392 raft_consensus.cc:695] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:47.562790 1392 consensus_queue.cc:237] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.562994 1166 ts_tablet_manager.cc:1905] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.579942 1160 ts_tablet_manager.cc:1905] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.531606 1158 tablet_bootstrap.cc:654] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.536290 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6716e7428ce240fca7ca5b02f564a5df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.539217 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b47ee787ca4dc5a4a56ae582d15b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.539855 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a54393ff3264c52bfdeee2cb9046828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.546833 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b47ee787ca4dc5a4a56ae582d15b8b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:47.586268 776 raft_consensus.cc:3054] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.551359 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1589b79a56148b6ab2946943de39fb2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:47.589378 990 leader_election.cc:343] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:47.589558 990 leader_election.cc:304] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:47.590775 776 raft_consensus.cc:2463] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
W20250109 20:52:47.592123 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6acca65ecee94f8cbe3f2b89e744aa6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6acca65ecee94f8cbe3f2b89e744aa6c already in progress: creating tablet
I20250109 20:52:47.592262 1158 tablet_bootstrap.cc:492] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.592450 1158 ts_tablet_manager.cc:1397] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.002s
I20250109 20:52:47.593760 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.594022 740 tablet_service.cc:1514] Processing DeleteTablet for tablet ca816507d0c047668a51f6e7ebce18cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce0a941f0c14501a947a5c7e2a844e8) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.594282 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.594363 1176 ts_tablet_manager.cc:1428] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.076s user 0.000s sys 0.018s
I20250109 20:52:47.593994 1158 raft_consensus.cc:357] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.594825 1158 raft_consensus.cc:383] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.594877 1158 raft_consensus.cc:738] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.595088 1158 consensus_queue.cc:260] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.595119 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2307b80fa70945568bfb4848975275b0" } new_table_name: "default.Table_3f42577ce23e4ed6b0f6ef3d41a6f5d0" modify_external_catalogs: true
I20250109 20:52:47.595630 1158 ts_tablet_manager.cc:1428] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250109 20:52:47.595685 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bc52c54bc34a44d88aab13982fc4e59b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.596351 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc52c54bc34a44d88aab13982fc4e59b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.597267 1158 tablet_bootstrap.cc:492] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.597054 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1eee2fc76a024644bdd78f7fd1ef0e62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.597756 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eee2fc76a024644bdd78f7fd1ef0e62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.598819 1158 tablet_bootstrap.cc:654] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.598784 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_52be79c241714f32810d3a84b9b0e36f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.599411 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52be79c241714f32810d3a84b9b0e36f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.600914 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c6f02c8ee12943ba91552c8fa3eb5118" } new_table_name: "default.Table_6640cef9559a4ba18d06c0aed445852f" modify_external_catalogs: true
I20250109 20:52:47.600629 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b590793190454bb09e3e7dd36515f65a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.601405 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b590793190454bb09e3e7dd36515f65a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.603827 1157 tablet_replica.cc:331] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.603729 1456 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:47.594563 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.600035 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.602383 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet ca816507d0c047668a51f6e7ebce18cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce0a941f0c14501a947a5c7e2a844e8) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.608389 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d5db66989ced4d6fb84352f19e087cc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.610278 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5db66989ced4d6fb84352f19e087cc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.604383 1157 raft_consensus.cc:2238] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.613029 1157 raft_consensus.cc:2267] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.613823 1157 ts_tablet_manager.cc:1905] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.594552 1451 raft_consensus.cc:2743] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:47.615906 1165 tablet_bootstrap.cc:492] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.616259 1165 ts_tablet_manager.cc:1397] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.126s user 0.007s sys 0.002s
I20250109 20:52:47.617256 1165 raft_consensus.cc:357] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.615078 1457 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250109 20:52:47.619822 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2d96a569e6be4e01a26918d115cfbcf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d96a569e6be4e01a26918d115cfbcf9 already in progress: creating tablet
W20250109 20:52:47.620077 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 49c6f1833ff1453f90a6ec6435248a02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49c6f1833ff1453f90a6ec6435248a02 already in progress: creating tablet
I20250109 20:52:47.604504 1176 tablet_bootstrap.cc:492] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:47.620551 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.620885 1408 mvcc.cc:204] Tried to move back new op lower bound from 7112523642879156224 to 7112523639501766656. Current Snapshot: MvccSnapshot[applied={T|T < 7112523642879156224}]
I20250109 20:52:47.622099 1165 raft_consensus.cc:383] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.622256 1165 raft_consensus.cc:738] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.622560 1165 consensus_queue.cc:260] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.622762 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6acadf60d566407aabc1a2395d1d235f (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]) successfully deleted
I20250109 20:52:47.623189 1165 ts_tablet_manager.cc:1428] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:52:47.623525 1165 tablet_bootstrap.cc:492] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.627776 1176 tablet_bootstrap.cc:654] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.618155 1166 ts_tablet_manager.cc:1918] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.628325 1166 log.cc:1198] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2a54393ff3264c52bfdeee2cb9046828
I20250109 20:52:47.630544 1166 ts_tablet_manager.cc:1939] T 2a54393ff3264c52bfdeee2cb9046828 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:47.631150 1165 tablet_bootstrap.cc:654] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.631352 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 294) took 1178 ms. Trace:
I20250109 20:52:47.631474 1166 rpcz_store.cc:276] 0109 20:52:46.452761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.452873 (+ 112us) service_pool.cc:225] Handling call
0109 20:52:47.560840 (+1107967us) ts_tablet_manager.cc:1145] Deleting tablet 2a54393ff3264c52bfdeee2cb9046828
0109 20:52:47.560855 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.573196 (+ 12341us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:47.617986 (+ 44790us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:47.631316 (+ 13330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107207}
I20250109 20:52:47.631922 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2a54393ff3264c52bfdeee2cb9046828 (table default.Table_0504385486ba4dd18a9a56a3f206aa9a [id=a79a90b8e83e4b4a80cb0e2a8693d7f6]) successfully deleted
I20250109 20:52:47.634127 1160 ts_tablet_manager.cc:1918] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.634246 1160 log.cc:1198] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2a54393ff3264c52bfdeee2cb9046828
I20250109 20:52:47.634622 1160 ts_tablet_manager.cc:1939] T 2a54393ff3264c52bfdeee2cb9046828 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.635080 1160 tablet_replica.cc:331] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.635156 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170]). Expected version 2 got 0
I20250109 20:52:47.635259 1160 raft_consensus.cc:2238] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.635248 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2a54393ff3264c52bfdeee2cb9046828 (table default.Table_0504385486ba4dd18a9a56a3f206aa9a [id=a79a90b8e83e4b4a80cb0e2a8693d7f6]) successfully deleted
I20250109 20:52:47.635406 1160 raft_consensus.cc:2267] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.637504 455 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170]). Expected version 2 got 0
I20250109 20:52:47.638477 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_eff4d27a17b3425c92c3d388ee1cc73e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.638906 1157 ts_tablet_manager.cc:1918] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.639027 1157 log.cc:1198] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f865acd2c9aa47ec8ebc3bf90d81c21e
W20250109 20:52:47.639017 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eff4d27a17b3425c92c3d388ee1cc73e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.639331 1157 ts_tablet_manager.cc:1939] T f865acd2c9aa47ec8ebc3bf90d81c21e P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.640588 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f865acd2c9aa47ec8ebc3bf90d81c21e (table default.Table_82b8503333e64f94829491f22a40b92b [id=87109c5773c14f159f596b9d30844948]) successfully deleted
I20250109 20:52:47.647269 1448 raft_consensus.cc:491] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.647415 1448 raft_consensus.cc:513] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.648062 1448 leader_election.cc:290] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:47.648144 1166 tablet_replica.cc:331] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:47.648830 1166 raft_consensus.cc:2238] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:47.649142 713 leader_election.cc:343] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2a54393ff3264c52bfdeee2cb9046828
I20250109 20:52:47.649411 1460 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:47.650848 1160 ts_tablet_manager.cc:1905] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.652743 1166 raft_consensus.cc:2267] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.653821 1158 tablet_bootstrap.cc:492] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.653960 1158 ts_tablet_manager.cc:1397] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.002s
I20250109 20:52:47.654822 1158 raft_consensus.cc:357] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.655005 1158 raft_consensus.cc:383] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.655081 1158 raft_consensus.cc:738] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.655292 1158 consensus_queue.cc:260] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.656287 1158 ts_tablet_manager.cc:1428] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:47.656553 1158 tablet_bootstrap.cc:492] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:47.659904 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 46d4cad7594346dfafdee8e4c49d74e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46d4cad7594346dfafdee8e4c49d74e6 already in progress: creating tablet
W20250109 20:52:47.660923 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 73e82535780c47ca8b9cd94b013b4b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73e82535780c47ca8b9cd94b013b4b42 already in progress: creating tablet
W20250109 20:52:47.662005 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6716e7428ce240fca7ca5b02f564a5df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6716e7428ce240fca7ca5b02f564a5df already in progress: creating tablet
I20250109 20:52:47.662631 1157 tablet_replica.cc:331] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.663198 1157 raft_consensus.cc:2238] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.663810 1157 raft_consensus.cc:2267] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:47.665957 712 leader_election.cc:343] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Not found: Tablet not found: 2a54393ff3264c52bfdeee2cb9046828
I20250109 20:52:47.666381 712 leader_election.cc:304] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:47.666795 1268 raft_consensus.cc:2743] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:47.667701 1157 ts_tablet_manager.cc:1905] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.672006 1158 tablet_bootstrap.cc:654] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.679629 1160 ts_tablet_manager.cc:1918] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:47.679879 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82209a6b39bb48e791ff63a8167c9765 has no consensus state
I20250109 20:52:47.680815 1157 ts_tablet_manager.cc:1918] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.680930 1157 log.cc:1198] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/03b47ee787ca4dc5a4a56ae582d15b8b
I20250109 20:52:47.681870 1157 ts_tablet_manager.cc:1939] T 03b47ee787ca4dc5a4a56ae582d15b8b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.682651 744 tablet_service.cc:1467] Processing CreateTablet for tablet d3681efa8785480ca65a51b4b40448f1 (DEFAULT_TABLE table=default.Table_3349fa7d981e4c0cbaea5fce12ad7c67 [id=6e40bdc674a9444181bd7deacfdea27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.682837 1157 tablet_replica.cc:331] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.683049 1157 raft_consensus.cc:2238] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.683107 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3681efa8785480ca65a51b4b40448f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.683240 1157 raft_consensus.cc:2267] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.683642 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet ed4866f6007d411da5cf4832fa92ece9 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]). Expected version 2 got 0
I20250109 20:52:47.683687 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170]). Expected version 2 got 0
I20250109 20:52:47.684330 450 master_service.cc:946] ReplaceTablet: received request to replace tablet ed4866f6007d411da5cf4832fa92ece9 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.686969 1157 ts_tablet_manager.cc:1905] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.688084 1166 ts_tablet_manager.cc:1905] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.689008 1032 tablet_service.cc:1467] Processing CreateTablet for tablet d3681efa8785480ca65a51b4b40448f1 (DEFAULT_TABLE table=default.Table_3349fa7d981e4c0cbaea5fce12ad7c67 [id=6e40bdc674a9444181bd7deacfdea27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.689425 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3681efa8785480ca65a51b4b40448f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.696308 740 tablet_service.cc:1467] Processing CreateTablet for tablet 8539cefebb844b3098a308ca580b0e92 (DEFAULT_TABLE table=default.Table_1d52e3d34dc14e729a515dc24e0787d4 [id=b74c21f2cb874f2485c0e3bc1d6e0dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.696724 742 tablet_service.cc:1467] Processing CreateTablet for tablet bba303032679462c8b16f706096b6d4d (DEFAULT_TABLE table=default.Table_fdc4d53d46874cb0a25cd71dfa7a3960 [id=6a25ad9a50db4cd7b62f73b4e268fdae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.696743 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8539cefebb844b3098a308ca580b0e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.700608 1157 ts_tablet_manager.cc:1918] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.700769 1157 log.cc:1198] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2a54393ff3264c52bfdeee2cb9046828
I20250109 20:52:47.700920 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bba303032679462c8b16f706096b6d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.701789 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3af2a00ab04b41915cffc3eb9206bc (DEFAULT_TABLE table=default.Table_87926cb29c4d4a6483062503677f2798 [id=ffee48c2614543b981c241ea4a3dc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.702178 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3af2a00ab04b41915cffc3eb9206bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.705260 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3af2a00ab04b41915cffc3eb9206bc (DEFAULT_TABLE table=default.Table_87926cb29c4d4a6483062503677f2798 [id=ffee48c2614543b981c241ea4a3dc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.705611 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3af2a00ab04b41915cffc3eb9206bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.706420 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_14ad6cd05135486e87607f62e9dd62c4" } modify_external_catalogs: true
I20250109 20:52:47.706571 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_14ad6cd05135486e87607f62e9dd62c4" } modify_external_catalogs: true
I20250109 20:52:47.707870 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a5bb2f6d26e3423e93672a1954c238a7" } modify_external_catalogs: true
I20250109 20:52:47.708055 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a5bb2f6d26e3423e93672a1954c238a7" } modify_external_catalogs: true
I20250109 20:52:47.709117 1157 ts_tablet_manager.cc:1939] T 2a54393ff3264c52bfdeee2cb9046828 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.709265 1033 tablet_service.cc:1467] Processing CreateTablet for tablet bba303032679462c8b16f706096b6d4d (DEFAULT_TABLE table=default.Table_fdc4d53d46874cb0a25cd71dfa7a3960 [id=6a25ad9a50db4cd7b62f73b4e268fdae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.709602 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bba303032679462c8b16f706096b6d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.709743 453 master_service.cc:946] ReplaceTablet: received request to replace tablet c528a1a9ae0541ba9d4a80e38e743969 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.715447 742 tablet_service.cc:1467] Processing CreateTablet for tablet 225046d45c1c43b488babf5da8de75d7 (DEFAULT_TABLE table=default.Table_711789d44eb242309002efa5bd06ffb1 [id=a638ba7e8a3b4c309df4fd05ca948518]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.715816 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225046d45c1c43b488babf5da8de75d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.723924 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_8d84afcb90094c7fa27ee7102fb5088d [id=53718e55e4344456851f408909cc8170] alter complete (version 2)
I20250109 20:52:47.724699 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 6acca65ecee94f8cbe3f2b89e744aa6c from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:47.724992 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ed134863bf642e79c26e5bf75b248b0 consensus state has no leader
I20250109 20:52:47.726101 1158 tablet_bootstrap.cc:492] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.726294 1158 ts_tablet_manager.cc:1397] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20250109 20:52:47.727260 1158 raft_consensus.cc:357] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.727443 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a0d09daa2eed41c3952fa21aa2cdd09e
I20250109 20:52:47.728065 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ed4866f6007d411da5cf4832fa92ece9
I20250109 20:52:47.728185 1158 raft_consensus.cc:383] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.728268 1158 raft_consensus.cc:738] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.728456 450 catalog_manager.cc:6596] ReplaceTablet: tablet ed4866f6007d411da5cf4832fa92ece9 deleted and replaced by tablet da7f67026b164e0faad128d34188afb5
I20250109 20:52:47.728508 1158 consensus_queue.cc:260] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.729774 1158 ts_tablet_manager.cc:1428] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250109 20:52:47.729828 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c528a1a9ae0541ba9d4a80e38e743969
I20250109 20:52:47.730120 1158 tablet_bootstrap.cc:492] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.730249 453 catalog_manager.cc:6596] ReplaceTablet: tablet c528a1a9ae0541ba9d4a80e38e743969 deleted and replaced by tablet e716270ffcf44039b075023fd0c2d2a3
I20250109 20:52:47.732576 1158 tablet_bootstrap.cc:654] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.734100 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 03b47ee787ca4dc5a4a56ae582d15b8b (table default.Table_51da7d20722144bf824bdc0388d68808 [id=92af12dca25846f694464bd77a74c1e3]) successfully deleted
I20250109 20:52:47.734131 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 225046d45c1c43b488babf5da8de75d7 (DEFAULT_TABLE table=default.Table_711789d44eb242309002efa5bd06ffb1 [id=a638ba7e8a3b4c309df4fd05ca948518]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.734577 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225046d45c1c43b488babf5da8de75d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.734826 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2a54393ff3264c52bfdeee2cb9046828 (table default.Table_0504385486ba4dd18a9a56a3f206aa9a [id=a79a90b8e83e4b4a80cb0e2a8693d7f6]) successfully deleted
W20250109 20:52:47.735066 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 already in progress: creating tablet
W20250109 20:52:47.735235 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ca816507d0c047668a51f6e7ebce18cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca816507d0c047668a51f6e7ebce18cb already in progress: creating tablet
W20250109 20:52:47.735405 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b68d23fb2df743f4afa2682f6d2e3efa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b68d23fb2df743f4afa2682f6d2e3efa already in progress: creating tablet
I20250109 20:52:47.737947 753 tablet_service.cc:1467] Processing CreateTablet for tablet 1c65489a602141239c99e8e8dae22ab4 (DEFAULT_TABLE table=default.Table_bd7a370b6cc2487da8595058ef89e0f5 [id=2d996a757dae433a9853245f2ceb1f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.738332 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c65489a602141239c99e8e8dae22ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.738948 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.739202 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.739367 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c528a1a9ae0541ba9d4a80e38e743969 with delete_type TABLET_DATA_DELETED (replaced by tablet e716270ffcf44039b075023fd0c2d2a3) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.739566 1157 tablet_replica.cc:331] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.739742 1157 raft_consensus.cc:2238] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.739912 1157 raft_consensus.cc:2267] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.740705 1157 ts_tablet_manager.cc:1905] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.743633 742 tablet_service.cc:1467] Processing CreateTablet for tablet 85a95a31468b431fb4304cf6680016da (DEFAULT_TABLE table=default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.744043 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a95a31468b431fb4304cf6680016da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.749356 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e740022397a8451e830a159fa021e461" } modify_external_catalogs: true
I20250109 20:52:47.749696 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e740022397a8451e830a159fa021e461" } modify_external_catalogs: true
I20250109 20:52:47.760116 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet da2d6dda69da4a59b055106c1041ec2b (table default.Table_2ec8f8fde48b4f38881f6209ee4f4bac [id=ce415eef7536445b99cab5e0769d359b]). Expected version 1 got 0
I20250109 20:52:47.686080 895 tablet_service.cc:1467] Processing CreateTablet for tablet 8539cefebb844b3098a308ca580b0e92 (DEFAULT_TABLE table=default.Table_1d52e3d34dc14e729a515dc24e0787d4 [id=b74c21f2cb874f2485c0e3bc1d6e0dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.760636 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8539cefebb844b3098a308ca580b0e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.761413 740 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4866f6007d411da5cf4832fa92ece9 with delete_type TABLET_DATA_DELETED (replaced by tablet da7f67026b164e0faad128d34188afb5) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.763056 1157 ts_tablet_manager.cc:1918] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.763154 1157 log.cc:1198] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/3fa23057a60a4a3eb7755e799bc3b911
I20250109 20:52:47.763495 1157 ts_tablet_manager.cc:1939] T 3fa23057a60a4a3eb7755e799bc3b911 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:47.764492 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 3fa23057a60a4a3eb7755e799bc3b911 (table default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee [id=00e9e64762824d89b0a757e997078b67]) successfully deleted
I20250109 20:52:47.686245 891 tablet_service.cc:1467] Processing CreateTablet for tablet d3681efa8785480ca65a51b4b40448f1 (DEFAULT_TABLE table=default.Table_3349fa7d981e4c0cbaea5fce12ad7c67 [id=6e40bdc674a9444181bd7deacfdea27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.686327 1191 raft_consensus.cc:491] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.765045 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3681efa8785480ca65a51b4b40448f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.764897 1191 raft_consensus.cc:513] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.765558 1191 leader_election.cc:290] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:47.689388 899 tablet_service.cc:1467] Processing CreateTablet for tablet bba303032679462c8b16f706096b6d4d (DEFAULT_TABLE table=default.Table_fdc4d53d46874cb0a25cd71dfa7a3960 [id=6a25ad9a50db4cd7b62f73b4e268fdae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.766546 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bba303032679462c8b16f706096b6d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.768592 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4f020c5ba643aeb11f8e08d748be46" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:47.689671 897 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3af2a00ab04b41915cffc3eb9206bc (DEFAULT_TABLE table=default.Table_87926cb29c4d4a6483062503677f2798 [id=ffee48c2614543b981c241ea4a3dc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.769038 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3af2a00ab04b41915cffc3eb9206bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:47.773423 854 leader_election.cc:343] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:47.689988 888 tablet_service.cc:1467] Processing CreateTablet for tablet 225046d45c1c43b488babf5da8de75d7 (DEFAULT_TABLE table=default.Table_711789d44eb242309002efa5bd06ffb1 [id=a638ba7e8a3b4c309df4fd05ca948518]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.690258 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1c65489a602141239c99e8e8dae22ab4 (DEFAULT_TABLE table=default.Table_bd7a370b6cc2487da8595058ef89e0f5 [id=2d996a757dae433a9853245f2ceb1f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.690819 893 tablet_service.cc:1467] Processing CreateTablet for tablet 85a95a31468b431fb4304cf6680016da (DEFAULT_TABLE table=default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.727627 886 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.728391 887 tablet_service.cc:1514] Processing DeleteTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.730090 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4866f6007d411da5cf4832fa92ece9 with delete_type TABLET_DATA_DELETED (replaced by tablet da7f67026b164e0faad128d34188afb5) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.731297 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c528a1a9ae0541ba9d4a80e38e743969 with delete_type TABLET_DATA_DELETED (replaced by tablet e716270ffcf44039b075023fd0c2d2a3) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.759032 1165 tablet_bootstrap.cc:492] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.774318 1165 ts_tablet_manager.cc:1397] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.008s
I20250109 20:52:47.774847 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d044013ec92a4c9186309687a275377e" } new_table_name: "default.Table_4b61b24f5f024f8094ea0c19aeea208a" modify_external_catalogs: true
I20250109 20:52:47.775206 1165 raft_consensus.cc:357] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.775434 1165 raft_consensus.cc:383] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.775542 1165 raft_consensus.cc:738] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.775810 1165 consensus_queue.cc:260] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.776372 1165 ts_tablet_manager.cc:1428] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:47.776685 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1d1722af0fbc4d2faa77942c00d9d3d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.777269 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d1722af0fbc4d2faa77942c00d9d3d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.778249 1160 log.cc:1198] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/6acadf60d566407aabc1a2395d1d235f
I20250109 20:52:47.778659 1160 ts_tablet_manager.cc:1939] T 6acadf60d566407aabc1a2395d1d235f P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.778673 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e46d69f1ef0344d591706192682b73c9" } new_table_name: "default.Table_76ad881a7d784b1a8eabb97a35b2d13e" modify_external_catalogs: true
I20250109 20:52:47.779762 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_68ffdf1aeefe420ea24aa1102012f77c" } modify_external_catalogs: true
I20250109 20:52:47.779907 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_68ffdf1aeefe420ea24aa1102012f77c" } modify_external_catalogs: true
I20250109 20:52:47.780978 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e4f018070aac4f4298a611b89b0e5f6a" } new_table_name: "default.Table_f53efacb818d49bea258bd79021056d3" modify_external_catalogs: true
I20250109 20:52:47.782421 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bbb3e86398ca4302b3f758d9a36837f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.782972 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbb3e86398ca4302b3f758d9a36837f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.784787 1158 tablet_bootstrap.cc:492] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:47.785029 1158 ts_tablet_manager.cc:1397] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.055s user 0.022s sys 0.020s
I20250109 20:52:47.786279 1158 raft_consensus.cc:357] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.786758 1158 raft_consensus.cc:383] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.786921 1158 raft_consensus.cc:738] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.787370 1158 consensus_queue.cc:260] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.788447 1158 ts_tablet_manager.cc:1428] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:47.789495 1165 tablet_bootstrap.cc:492] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.789530 1158 tablet_bootstrap.cc:492] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:47.792539 1158 tablet_bootstrap.cc:654] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.793354 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 6acadf60d566407aabc1a2395d1d235f (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]) successfully deleted
I20250109 20:52:47.793411 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225046d45c1c43b488babf5da8de75d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:47.793666 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 46d4cad7594346dfafdee8e4c49d74e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46d4cad7594346dfafdee8e4c49d74e6 already in progress: creating tablet
W20250109 20:52:47.793917 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 73e82535780c47ca8b9cd94b013b4b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73e82535780c47ca8b9cd94b013b4b42 already in progress: creating tablet
I20250109 20:52:47.796291 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1925a82eb711409ab2c5429c61edf9d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.798799 1160 tablet_replica.cc:331] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:47.799052 1160 raft_consensus.cc:2238] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.799438 1160 raft_consensus.cc:2267] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.801354 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ef823ba5464d429db5aa6b8c3ccc1934" } modify_external_catalogs: true
I20250109 20:52:47.801884 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ef823ba5464d429db5aa6b8c3ccc1934" } modify_external_catalogs: true
I20250109 20:52:47.804517 1165 tablet_bootstrap.cc:654] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.805497 1160 ts_tablet_manager.cc:1905] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.805955 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a95a31468b431fb4304cf6680016da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.809121 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c65489a602141239c99e8e8dae22ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.804634 744 tablet_service.cc:1514] Processing DeleteTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.821748 1166 ts_tablet_manager.cc:1918] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.736490 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.737151 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet c528a1a9ae0541ba9d4a80e38e743969 with delete_type TABLET_DATA_DELETED (replaced by tablet e716270ffcf44039b075023fd0c2d2a3) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.738420 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4866f6007d411da5cf4832fa92ece9 with delete_type TABLET_DATA_DELETED (replaced by tablet da7f67026b164e0faad128d34188afb5) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.740979 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8539cefebb844b3098a308ca580b0e92 (DEFAULT_TABLE table=default.Table_1d52e3d34dc14e729a515dc24e0787d4 [id=b74c21f2cb874f2485c0e3bc1d6e0dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.822366 1166 log.cc:1198] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6acadf60d566407aabc1a2395d1d235f
I20250109 20:52:47.755154 1176 tablet_bootstrap.cc:492] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:47.803437 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4f020c5ba643aeb11f8e08d748be46" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:47.736315 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 85a95a31468b431fb4304cf6680016da (DEFAULT_TABLE table=default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.822880 1176 ts_tablet_manager.cc:1397] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.218s user 0.010s sys 0.002s
I20250109 20:52:47.822434 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8539cefebb844b3098a308ca580b0e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.823386 1157 tablet_replica.cc:331] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:47.823629 1157 raft_consensus.cc:2238] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:47.823827 1157 raft_consensus.cc:2267] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:47.823940 1176 raft_consensus.cc:357] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.824204 1176 raft_consensus.cc:383] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.824342 1176 raft_consensus.cc:738] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.824734 1176 consensus_queue.cc:260] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.825356 1176 ts_tablet_manager.cc:1428] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:47.825536 1157 ts_tablet_manager.cc:1905] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:47.825690 1176 tablet_bootstrap.cc:492] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:47.827513 1176 tablet_bootstrap.cc:654] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.750432 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 1c65489a602141239c99e8e8dae22ab4 (DEFAULT_TABLE table=default.Table_bd7a370b6cc2487da8595058ef89e0f5 [id=2d996a757dae433a9853245f2ceb1f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.828891 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c65489a602141239c99e8e8dae22ab4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:47.835594 854 leader_election.cc:343] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:47.835793 854 leader_election.cc:304] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:47.837496 1191 raft_consensus.cc:2743] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:47.842173 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a95a31468b431fb4304cf6680016da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.857101 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b77c8d6adbf44375aff2782a83643255
I20250109 20:52:47.860069 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7ab416cb67944fa5b70140a8b8ab5e45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.860560 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ab416cb67944fa5b70140a8b8ab5e45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.861177 899 tablet_service.cc:1514] Processing DeleteTablet for tablet b77c8d6adbf44375aff2782a83643255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.873745 1157 ts_tablet_manager.cc:1918] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:47.877501 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ed7abf23ce04a268a460d8e3e97d56b has no consensus state
W20250109 20:52:47.880146 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70b48900f57f47128e31b27275b1489c consensus state has no leader
I20250109 20:52:47.881057 1166 ts_tablet_manager.cc:1939] T 6acadf60d566407aabc1a2395d1d235f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:47.882277 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b77c8d6adbf44375aff2782a83643255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.883766 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 295) took 1404 ms. Trace:
I20250109 20:52:47.888953 1165 tablet_bootstrap.cc:492] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:47.889358 1165 ts_tablet_manager.cc:1397] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.002s
I20250109 20:52:47.889268 1166 rpcz_store.cc:276] 0109 20:52:46.479178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.479258 (+ 80us) service_pool.cc:225] Handling call
0109 20:52:47.648050 (+1168792us) ts_tablet_manager.cc:1145] Deleting tablet 6acadf60d566407aabc1a2395d1d235f
0109 20:52:47.648065 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.755343 (+107278us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:47.821598 (+ 66255us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:47.883683 (+ 62085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1168037}
I20250109 20:52:47.889919 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 305) took 1300 ms. Trace:
I20250109 20:52:47.890100 1166 rpcz_store.cc:276] 0109 20:52:46.589060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.589161 (+ 101us) service_pool.cc:225] Handling call
0109 20:52:47.889696 (+1300535us) ts_tablet_manager.cc:1145] Deleting tablet ed5ea5d588834f1b8b2b057ec324aadc
0109 20:52:47.889724 (+ 28us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.889856 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299264}
I20250109 20:52:47.890352 1165 raft_consensus.cc:357] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.890700 1165 raft_consensus.cc:383] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.890848 1165 raft_consensus.cc:738] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.891048 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6acadf60d566407aabc1a2395d1d235f (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]) successfully deleted
I20250109 20:52:47.891217 1165 consensus_queue.cc:260] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:47.891371 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ed5ea5d588834f1b8b2b057ec324aadc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed5ea5d588834f1b8b2b057ec324aadc already in progress: creating tablet
I20250109 20:52:47.894606 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.896984 451 master_service.cc:946] ReplaceTablet: received request to replace tablet dae574bf93d649099252e9e7b09b748c from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:47.897127 1165 ts_tablet_manager.cc:1428] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
W20250109 20:52:47.898335 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b31b9785ee21418ba13b426ba94cfa9b consensus state has no leader
I20250109 20:52:47.899277 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f1c5a6b9681d4ef6a36e308ab2916423
I20250109 20:52:47.900094 888 tablet_service.cc:1514] Processing DeleteTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:47.890873 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 307) took 1280 ms. Trace:
I20250109 20:52:47.902858 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b1cfdfcec71b4de6b182f4598f9e2601" } new_table_name: "default.Table_5c9e541186c64e158515aff3e19ccda9" modify_external_catalogs: true
I20250109 20:52:47.902240 1166 rpcz_store.cc:276] 0109 20:52:46.610559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.610612 (+ 53us) service_pool.cc:225] Handling call
0109 20:52:47.890722 (+1280110us) ts_tablet_manager.cc:1145] Deleting tablet 34f215eee55d4e6a97a384df6ff9d596
0109 20:52:47.890734 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.890828 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279765}
I20250109 20:52:47.903400 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 308) took 1292 ms. Trace:
I20250109 20:52:47.903581 1166 rpcz_store.cc:276] 0109 20:52:46.611188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.611232 (+ 44us) service_pool.cc:225] Handling call
0109 20:52:47.903209 (+1291977us) ts_tablet_manager.cc:1145] Deleting tablet 2b47656fe1ae4c9ba4138bcd37c0e6db
0109 20:52:47.903221 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.903344 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291806}
I20250109 20:52:47.904286 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 309) took 1252 ms. Trace:
I20250109 20:52:47.903353 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_986f5a94df1245f7930aca6efd8614a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.904454 1166 rpcz_store.cc:276] 0109 20:52:46.651934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.652944 (+ 1010us) service_pool.cc:225] Handling call
0109 20:52:47.904117 (+1251173us) ts_tablet_manager.cc:1145] Deleting tablet ff4f020c5ba643aeb11f8e08d748be46
0109 20:52:47.904130 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.904230 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1250937}
I20250109 20:52:47.905072 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 313) took 1173 ms. Trace:
I20250109 20:52:47.905256 1166 rpcz_store.cc:276] 0109 20:52:46.731877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.731950 (+ 73us) service_pool.cc:225] Handling call
0109 20:52:47.904898 (+1172948us) ts_tablet_manager.cc:1145] Deleting tablet a8da9781ab32429297d0c875c1971501
0109 20:52:47.904909 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.905012 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142335}
I20250109 20:52:47.905833 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 314) took 1173 ms. Trace:
I20250109 20:52:47.906013 1166 rpcz_store.cc:276] 0109 20:52:46.732208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.732255 (+ 47us) service_pool.cc:225] Handling call
0109 20:52:47.905666 (+1173411us) ts_tablet_manager.cc:1145] Deleting tablet bb0d889cb1114edc8b799b41b23a54a6
0109 20:52:47.905676 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.905782 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141407}
W20250109 20:52:47.906426 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 34f215eee55d4e6a97a384df6ff9d596 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34f215eee55d4e6a97a384df6ff9d596 already in progress: creating tablet
W20250109 20:52:47.906765 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b47656fe1ae4c9ba4138bcd37c0e6db already in progress: creating tablet
W20250109 20:52:47.907056 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ff4f020c5ba643aeb11f8e08d748be46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4f020c5ba643aeb11f8e08d748be46 already in progress: creating tablet
W20250109 20:52:47.907292 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a8da9781ab32429297d0c875c1971501 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8da9781ab32429297d0c875c1971501 already in progress: creating tablet
I20250109 20:52:47.907662 1165 tablet_bootstrap.cc:492] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:47.909888 1165 tablet_bootstrap.cc:654] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:47.908702 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_986f5a94df1245f7930aca6efd8614a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.912535 1160 ts_tablet_manager.cc:1918] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:47.912904 1160 log.cc:1198] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/03b47ee787ca4dc5a4a56ae582d15b8b
I20250109 20:52:47.913311 1160 ts_tablet_manager.cc:1939] T 03b47ee787ca4dc5a4a56ae582d15b8b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:47.913728 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_06d47c4610b54cb7b1b655f6dc07db72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.914559 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bea81cd569304584b538f85a10ebc45b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.915094 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bea81cd569304584b538f85a10ebc45b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:47.916268 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06d47c4610b54cb7b1b655f6dc07db72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.917356 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6d708581ee5c4179af45f5e8ce12bec5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:47.917945 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d708581ee5c4179af45f5e8ce12bec5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.919348 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 03b47ee787ca4dc5a4a56ae582d15b8b (table default.Table_51da7d20722144bf824bdc0388d68808 [id=92af12dca25846f694464bd77a74c1e3]) successfully deleted
W20250109 20:52:47.920004 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet bb0d889cb1114edc8b799b41b23a54a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb0d889cb1114edc8b799b41b23a54a6 already in progress: creating tablet
I20250109 20:52:47.912771 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:47.920641 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 315) took 1188 ms. Trace:
W20250109 20:52:47.921294 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6716e7428ce240fca7ca5b02f564a5df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6716e7428ce240fca7ca5b02f564a5df already in progress: creating tablet
W20250109 20:52:47.922078 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 already in progress: creating tablet
I20250109 20:52:47.922317 1166 rpcz_store.cc:276] 0109 20:52:46.732530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.732581 (+ 51us) service_pool.cc:225] Handling call
0109 20:52:47.920469 (+1187888us) ts_tablet_manager.cc:1145] Deleting tablet deed945f2ecf4290ad002662c3f66030
0109 20:52:47.920480 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:47.920588 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1156156}
W20250109 20:52:47.929165 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ca816507d0c047668a51f6e7ebce18cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca816507d0c047668a51f6e7ebce18cb already in progress: creating tablet
W20250109 20:52:47.929857 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet deed945f2ecf4290ad002662c3f66030 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deed945f2ecf4290ad002662c3f66030 already in progress: creating tablet
I20250109 20:52:47.931797 893 tablet_service.cc:1467] Processing CreateTablet for tablet 0cede2a1882946a181b18d66f12c5114 (DEFAULT_TABLE table=default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.931911 888 tablet_service.cc:1467] Processing CreateTablet for tablet f03d4e679a1c4793bd83cd979acced88 (DEFAULT_TABLE table=default.Table_7d9ef737f1e74ae1b6f66968d5c0b586 [id=5fbc12ffb04546deb78133a5796c63c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.932272 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03d4e679a1c4793bd83cd979acced88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.932732 1451 raft_consensus.cc:491] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.932879 1451 raft_consensus.cc:513] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.933007 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 4c112bab5c5a42efab7582d83ca7eac1 (table default.Table_5894addb4e7a40a8b557929948171f8b [id=b7903ec3ae3b4d0f8ef509c4c05f11f1]). Expected version 5 got 0
I20250109 20:52:47.934093 1451 leader_election.cc:290] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:47.935639 899 tablet_service.cc:1467] Processing CreateTablet for tablet 82209a6b39bb48e791ff63a8167c9765 (DEFAULT_TABLE table=default.Table_3f42577ce23e4ed6b0f6ef3d41a6f5d0 [id=cced112727b0498f955953f2b2d29551]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.936075 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82209a6b39bb48e791ff63a8167c9765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.938788 895 tablet_service.cc:1467] Processing CreateTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 (DEFAULT_TABLE table=default.Table_49bb1f7179c6420c83e530237052e180 [id=38d4e8c8c6294d3983708d35f536e400]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.939182 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28cd9cd48910497ba3e4b159583cb7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.939237 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 4c112bab5c5a42efab7582d83ca7eac1 (table default.Table_5894addb4e7a40a8b557929948171f8b [id=b7903ec3ae3b4d0f8ef509c4c05f11f1]). Expected version 5 got 0
I20250109 20:52:47.940197 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (DEFAULT_TABLE table=default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.940622 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce0a941f0c14501a947a5c7e2a844e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.943331 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_03ed60a072e941b6b5b9a8a477aa9020"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.943683 892 tablet_service.cc:1467] Processing CreateTablet for tablet 5f953f43877c4a738ffd8255afbf8275 (DEFAULT_TABLE table=default.Table_19363d9282fe42228ab784489a2b9a17 [id=fa8fa63dbd44442b81a3b9640d42680e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.944010 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f953f43877c4a738ffd8255afbf8275. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:47.943993 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03ed60a072e941b6b5b9a8a477aa9020 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.950457 1451 raft_consensus.cc:491] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.950430 888 tablet_service.cc:1467] Processing CreateTablet for tablet 724d15e05a864852a118badae13a5d3d (DEFAULT_TABLE table=default.Table_b1cfdfcec71b4de6b182f4598f9e2601 [id=9fcc13c731ac492fbf382b79a3df9ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.950587 1451 raft_consensus.cc:513] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.950827 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d15e05a864852a118badae13a5d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.951279 1451 leader_election.cc:290] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:47.952579 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c528a1a9ae0541ba9d4a80e38e743969" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:47.953194 989 leader_election.cc:343] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:47.955780 897 tablet_service.cc:1467] Processing CreateTablet for tablet d57a8bbb3da84088a98c816102e4e9df (DEFAULT_TABLE table=default.Table_171f660e57e54000a542c3e8bcd0bfb5 [id=0bf46974692d47c38befe7eca4f23cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.955863 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9c94d3852ae945d0b0dd5798dac99fba" } new_table_name: "default.Table_261453a8462344cc9481a9b9c9165913" modify_external_catalogs: true
I20250109 20:52:47.956087 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57a8bbb3da84088a98c816102e4e9df. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:47.966100 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 724d15e05a864852a118badae13a5d3d consensus state has no leader
I20250109 20:52:47.968559 1191 raft_consensus.cc:491] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.968792 1191 raft_consensus.cc:513] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.969558 1191 leader_election.cc:290] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:47.969750 888 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b (DEFAULT_TABLE table=default.Table_4b61b24f5f024f8094ea0c19aeea208a [id=b296e8173f88419a8a14ac395611b4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.970460 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1925a82eb711409ab2c5429c61edf9d8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:47.970810 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4866f6007d411da5cf4832fa92ece9" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:47.970880 918 raft_consensus.cc:2463] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:47.971062 1054 raft_consensus.cc:2463] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:47.971470 854 leader_election.cc:304] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:47.971490 989 leader_election.cc:304] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:47.971889 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ed7abf23ce04a268a460d8e3e97d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.971987 1191 raft_consensus.cc:2798] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:47.972095 1191 raft_consensus.cc:491] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:47.972180 1191 raft_consensus.cc:3054] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.973074 1451 raft_consensus.cc:2798] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:47.973192 1451 raft_consensus.cc:491] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:47.973277 1451 raft_consensus.cc:3054] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.974604 1191 raft_consensus.cc:513] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.975277 1191 leader_election.cc:290] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:47.976940 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4866f6007d411da5cf4832fa92ece9" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:47.976884 1451 raft_consensus.cc:513] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.977138 1054 raft_consensus.cc:3054] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.979640 1054 raft_consensus.cc:2463] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:47.979823 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1925a82eb711409ab2c5429c61edf9d8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:47.980080 918 raft_consensus.cc:3054] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.976361 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:47.980520 854 leader_election.cc:304] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:47.980834 1191 raft_consensus.cc:2798] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election won for term 1
W20250109 20:52:47.980891 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:47.981789 1191 raft_consensus.cc:695] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Running, Role: LEADER
I20250109 20:52:47.982095 1191 consensus_queue.cc:237] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:47.983062 1483 raft_consensus.cc:491] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.983589 1176 tablet_bootstrap.cc:492] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:47.983809 1176 ts_tablet_manager.cc:1397] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.158s user 0.002s sys 0.025s
I20250109 20:52:47.983291 1483 raft_consensus.cc:513] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.984674 1176 raft_consensus.cc:357] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.984958 1176 raft_consensus.cc:383] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:47.985047 1176 raft_consensus.cc:738] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:47.985277 1176 consensus_queue.cc:260] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.985862 1176 ts_tablet_manager.cc:1428] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:52:47.986191 1176 tablet_bootstrap.cc:492] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:47.987929 1176 tablet_bootstrap.cc:654] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:47.979655 1451 leader_election.cc:290] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:47.991077 1033 tablet_service.cc:1467] Processing CreateTablet for tablet f03d4e679a1c4793bd83cd979acced88 (DEFAULT_TABLE table=default.Table_7d9ef737f1e74ae1b6f66968d5c0b586 [id=5fbc12ffb04546deb78133a5796c63c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.991400 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03d4e679a1c4793bd83cd979acced88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.897967 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b77c8d6adbf44375aff2782a83643255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.901454 740 tablet_service.cc:1514] Processing DeleteTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:47.930796 742 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (DEFAULT_TABLE table=default.Table_7dcbaf73ff544c3e8e804e28f8e20a6c [id=15c9f7d5d3ff4fa4865b55c5892421fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.934931 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1925a82eb711409ab2c5429c61edf9d8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:47.949420 1268 raft_consensus.cc:491] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:47.994920 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3c276fab6a4d068d64ea5d23c9a595. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:47.994828 1268 raft_consensus.cc:513] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:47.995433 1268 leader_election.cc:290] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:47.951987 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c528a1a9ae0541ba9d4a80e38e743969" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:47.970451 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4866f6007d411da5cf4832fa92ece9" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:47.995728 778 raft_consensus.cc:2463] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:47.978152 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1925a82eb711409ab2c5429c61edf9d8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:47.980093 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4866f6007d411da5cf4832fa92ece9" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:47.988679 753 tablet_service.cc:1467] Processing CreateTablet for tablet 0cede2a1882946a181b18d66f12c5114 (DEFAULT_TABLE table=default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.997149 775 raft_consensus.cc:3054] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:47.990288 751 tablet_service.cc:1467] Processing CreateTablet for tablet f03d4e679a1c4793bd83cd979acced88 (DEFAULT_TABLE table=default.Table_7d9ef737f1e74ae1b6f66968d5c0b586 [id=5fbc12ffb04546deb78133a5796c63c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.994115 741 tablet_service.cc:1467] Processing CreateTablet for tablet 82209a6b39bb48e791ff63a8167c9765 (DEFAULT_TABLE table=default.Table_3f42577ce23e4ed6b0f6ef3d41a6f5d0 [id=cced112727b0498f955953f2b2d29551]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:47.996254 990 leader_election.cc:343] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: Tablet not RUNNING: STOPPED
I20250109 20:52:47.994524 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0cede2a1882946a181b18d66f12c5114 (DEFAULT_TABLE table=default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:47.998546 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c112bab5c5a42efab7582d83ca7eac1" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
W20250109 20:52:47.999603 713 leader_election.cc:343] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.000792 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 5f953f43877c4a738ffd8255afbf8275 (DEFAULT_TABLE table=default.Table_19363d9282fe42228ab784489a2b9a17 [id=fa8fa63dbd44442b81a3b9640d42680e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.000844 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b (DEFAULT_TABLE table=default.Table_4b61b24f5f024f8094ea0c19aeea208a [id=b296e8173f88419a8a14ac395611b4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.001153 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 724d15e05a864852a118badae13a5d3d (DEFAULT_TABLE table=default.Table_b1cfdfcec71b4de6b182f4598f9e2601 [id=9fcc13c731ac492fbf382b79a3df9ed2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:48.002502 990 leader_election.cc:343] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.002637 990 leader_election.cc:304] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:48.002837 744 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (DEFAULT_TABLE table=default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:48.003489 990 leader_election.cc:343] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: Tablet not RUNNING: STOPPED
I20250109 20:52:48.002887 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (DEFAULT_TABLE table=default.Table_7dcbaf73ff544c3e8e804e28f8e20a6c [id=15c9f7d5d3ff4fa4865b55c5892421fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.003633 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 82209a6b39bb48e791ff63a8167c9765 (DEFAULT_TABLE table=default.Table_3f42577ce23e4ed6b0f6ef3d41a6f5d0 [id=cced112727b0498f955953f2b2d29551]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.004652 750 tablet_service.cc:1467] Processing CreateTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 (DEFAULT_TABLE table=default.Table_49bb1f7179c6420c83e530237052e180 [id=38d4e8c8c6294d3983708d35f536e400]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.005071 740 tablet_service.cc:1467] Processing CreateTablet for tablet d57a8bbb3da84088a98c816102e4e9df (DEFAULT_TABLE table=default.Table_171f660e57e54000a542c3e8bcd0bfb5 [id=0bf46974692d47c38befe7eca4f23cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.006129 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b (DEFAULT_TABLE table=default.Table_4b61b24f5f024f8094ea0c19aeea208a [id=b296e8173f88419a8a14ac395611b4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.009183 1451 raft_consensus.cc:2743] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:48.009557 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (DEFAULT_TABLE table=default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.009919 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 (DEFAULT_TABLE table=default.Table_49bb1f7179c6420c83e530237052e180 [id=38d4e8c8c6294d3983708d35f536e400]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.010206 1035 tablet_service.cc:1467] Processing CreateTablet for tablet d57a8bbb3da84088a98c816102e4e9df (DEFAULT_TABLE table=default.Table_171f660e57e54000a542c3e8bcd0bfb5 [id=0bf46974692d47c38befe7eca4f23cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.010463 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cede2a1882946a181b18d66f12c5114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.010886 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d15e05a864852a118badae13a5d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.011102 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57a8bbb3da84088a98c816102e4e9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.016410 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28cd9cd48910497ba3e4b159583cb7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.016942 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82209a6b39bb48e791ff63a8167c9765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.017349 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce0a941f0c14501a947a5c7e2a844e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.017556 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cede2a1882946a181b18d66f12c5114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.017798 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82209a6b39bb48e791ff63a8167c9765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.021149 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28cd9cd48910497ba3e4b159583cb7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.024948 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ed7abf23ce04a268a460d8e3e97d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.024953 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f953f43877c4a738ffd8255afbf8275. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.028877 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ed7abf23ce04a268a460d8e3e97d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.028910 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03d4e679a1c4793bd83cd979acced88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.031227 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3c276fab6a4d068d64ea5d23c9a595. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.032909 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce0a941f0c14501a947a5c7e2a844e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.040498 775 raft_consensus.cc:2463] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:48.040782 749 tablet_service.cc:1467] Processing CreateTablet for tablet 724d15e05a864852a118badae13a5d3d (DEFAULT_TABLE table=default.Table_b1cfdfcec71b4de6b182f4598f9e2601 [id=9fcc13c731ac492fbf382b79a3df9ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.041250 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d15e05a864852a118badae13a5d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.044793 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5f953f43877c4a738ffd8255afbf8275 (DEFAULT_TABLE table=default.Table_19363d9282fe42228ab784489a2b9a17 [id=fa8fa63dbd44442b81a3b9640d42680e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.045163 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f953f43877c4a738ffd8255afbf8275. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.049677 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57a8bbb3da84088a98c816102e4e9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.061300 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 316) took 1277 ms. Trace:
I20250109 20:52:48.061486 1166 rpcz_store.cc:276] 0109 20:52:46.784212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.784271 (+ 59us) service_pool.cc:225] Handling call
0109 20:52:48.061080 (+1276809us) ts_tablet_manager.cc:1145] Deleting tablet 5757b98100804c688f67e7111db2b622
0109 20:52:48.061096 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.061236 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1217791}
I20250109 20:52:48.062996 1166 tablet_replica.cc:331] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.066112 1166 raft_consensus.cc:2238] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:48.066538 1166 pending_rounds.cc:62] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Trying to abort 1 pending ops.
I20250109 20:52:48.067098 1166 pending_rounds.cc:69] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523643139837952 op_type: NO_OP noop_request { }
I20250109 20:52:48.067662 1166 raft_consensus.cc:2883] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:48.071327 1166 raft_consensus.cc:2267] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.072681 1157 log.cc:1198] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/1925a82eb711409ab2c5429c61edf9d8
I20250109 20:52:48.075838 1166 ts_tablet_manager.cc:1905] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.076689 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5894addb4e7a40a8b557929948171f8b" } modify_external_catalogs: true
I20250109 20:52:48.076927 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5894addb4e7a40a8b557929948171f8b" } modify_external_catalogs: true
I20250109 20:52:48.080363 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_64ebceb89d2c4e8291bc096091578453"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:48.082000 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64ebceb89d2c4e8291bc096091578453 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.084022 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_711789d44eb242309002efa5bd06ffb1" } modify_external_catalogs: true
I20250109 20:52:48.084295 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_711789d44eb242309002efa5bd06ffb1" } modify_external_catalogs: true
I20250109 20:52:48.085384 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_76ad881a7d784b1a8eabb97a35b2d13e" } new_table_name: "default.Table_cbde6546bcd3407ebd6470f02d881bdd" modify_external_catalogs: true
I20250109 20:52:48.086802 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cede2a1882946a181b18d66f12c5114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.090915 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.091006 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c112bab5c5a42efab7582d83ca7eac1" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:48.091162 1157 ts_tablet_manager.cc:1939] T 1925a82eb711409ab2c5429c61edf9d8 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:48.092051 1160 tablet_replica.cc:331] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:48.092566 1157 tablet_replica.cc:331] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:52:48.092661 712 leader_election.cc:343] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.092870 712 leader_election.cc:304] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:48.093329 1268 raft_consensus.cc:2743] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:48.093822 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5757b98100804c688f67e7111db2b622 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5757b98100804c688f67e7111db2b622 already in progress: creating tablet
I20250109 20:52:48.095538 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 1925a82eb711409ab2c5429c61edf9d8 (table default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]) successfully deleted
W20250109 20:52:48.095760 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c528a1a9ae0541ba9d4a80e38e743969 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c528a1a9ae0541ba9d4a80e38e743969 already in progress: creating tablet
I20250109 20:52:48.100661 1160 raft_consensus.cc:2238] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.100907 1160 raft_consensus.cc:2267] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.101658 1157 raft_consensus.cc:2238] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.101847 1157 raft_consensus.cc:2267] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.104235 1157 ts_tablet_manager.cc:1905] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.106611 1176 tablet_bootstrap.cc:492] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.107106 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (DEFAULT_TABLE table=default.Table_7dcbaf73ff544c3e8e804e28f8e20a6c [id=15c9f7d5d3ff4fa4865b55c5892421fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.107614 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3c276fab6a4d068d64ea5d23c9a595. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.108085 1176 ts_tablet_manager.cc:1397] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.032s
W20250109 20:52:48.107317 854 leader_election.cc:343] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 3fa23057a60a4a3eb7755e799bc3b911
I20250109 20:52:48.109340 1160 ts_tablet_manager.cc:1905] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.109320 1176 raft_consensus.cc:357] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.109515 1176 raft_consensus.cc:383] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.109602 1176 raft_consensus.cc:738] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.109812 1176 consensus_queue.cc:260] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.114161 1176 ts_tablet_manager.cc:1428] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250109 20:52:48.114809 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa23057a60a4a3eb7755e799bc3b911" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
W20250109 20:52:48.115463 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 -> Peer 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583): Couldn't send request to peer 36a29ab2002d4eb389cd11778988e1e9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250109 20:52:48.115612 1176 tablet_bootstrap.cc:492] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.118438 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2a89a0b47d10414d9b6a9123af1f4ab3
I20250109 20:52:48.118821 1176 tablet_bootstrap.cc:654] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.119127 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet dae574bf93d649099252e9e7b09b748c
I20250109 20:52:48.119520 451 catalog_manager.cc:6596] ReplaceTablet: tablet dae574bf93d649099252e9e7b09b748c deleted and replaced by tablet dc235614171f4a61b75492ddeb3f0fe6
I20250109 20:52:48.120020 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 402280f75e51463a865c0fb521d0fd8a
I20250109 20:52:48.120309 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_655ff82c5466442c818f93c7d408c157" } new_table_name: "default.Table_8c82f4521cc94881ad01252c01dd81b8" modify_external_catalogs: true
I20250109 20:52:48.121310 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3abc7e662927447b90d40c4c4ba6e04b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:48.121439 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c01a7fe9ae3947a59c88c37c98a59cab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:48.121842 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3abc7e662927447b90d40c4c4ba6e04b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:48.121974 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c01a7fe9ae3947a59c88c37c98a59cab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:48.123210 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9558dfe563d94482bf496e19b8a80edc consensus state has no leader
I20250109 20:52:48.123569 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8f9dfe1265284477bd928e1cc0c9e477" } new_table_name: "default.Table_86ed54bc2bc046709614f244faefef48" modify_external_catalogs: true
I20250109 20:52:48.123735 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_416c9e793e9d4fa69bd29e1e30365e06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:48.124171 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_416c9e793e9d4fa69bd29e1e30365e06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.125362 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d52320f2397e4526ae456263f71bcda2" } modify_external_catalogs: true
I20250109 20:52:48.125491 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d52320f2397e4526ae456263f71bcda2" } modify_external_catalogs: true
I20250109 20:52:48.127573 1483 leader_election.cc:290] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:48.129616 1158 tablet_bootstrap.cc:492] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:48.129772 1158 ts_tablet_manager.cc:1397] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.340s user 0.058s sys 0.150s
I20250109 20:52:48.130101 1176 tablet_bootstrap.cc:492] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.130069 918 raft_consensus.cc:2463] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:48.130607 1158 raft_consensus.cc:357] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.130795 1158 raft_consensus.cc:383] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.130864 1158 raft_consensus.cc:738] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.131104 1176 ts_tablet_manager.cc:1397] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20250109 20:52:48.131079 1158 consensus_queue.cc:260] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.131526 1158 ts_tablet_manager.cc:1428] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250109 20:52:48.132694 854 leader_election.cc:343] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.132652 1176 raft_consensus.cc:357] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.134888 1176 raft_consensus.cc:383] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.135218 1176 raft_consensus.cc:738] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.135605 1166 ts_tablet_manager.cc:1918] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:48.134737 854 leader_election.cc:304] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:48.135736 1166 log.cc:1198] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/03b47ee787ca4dc5a4a56ae582d15b8b
I20250109 20:52:48.135912 1176 consensus_queue.cc:260] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.137249 1176 ts_tablet_manager.cc:1428] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:48.137666 1158 tablet_bootstrap.cc:492] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:48.138710 1176 tablet_bootstrap.cc:492] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.141408 1176 tablet_bootstrap.cc:654] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.142175 1158 tablet_bootstrap.cc:654] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.145551 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.146039 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet dae574bf93d649099252e9e7b09b748c with delete_type TABLET_DATA_DELETED (replaced by tablet dc235614171f4a61b75492ddeb3f0fe6) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.146278 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.146553 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 402280f75e51463a865c0fb521d0fd8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.153354 1160 ts_tablet_manager.cc:1918] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.153494 1160 log.cc:1198] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ff4f020c5ba643aeb11f8e08d748be46
I20250109 20:52:48.153578 1451 raft_consensus.cc:491] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:48.153772 1451 raft_consensus.cc:513] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.153919 1160 ts_tablet_manager.cc:1939] T ff4f020c5ba643aeb11f8e08d748be46 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:48.154457 1451 leader_election.cc:290] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:48.154541 1160 tablet_replica.cc:331] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:48.136159 1166 ts_tablet_manager.cc:1939] T 03b47ee787ca4dc5a4a56ae582d15b8b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:48.156342 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b68d23fb2df743f4afa2682f6d2e3efa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b68d23fb2df743f4afa2682f6d2e3efa already in progress: creating tablet
I20250109 20:52:48.156698 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 323) took 1278 ms. Trace:
I20250109 20:52:48.157368 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ff4f020c5ba643aeb11f8e08d748be46 (table default.Table_aaff98763d0f4eff96db3f5d2d85ba2a [id=7bd35efd3fad42dab4f6aa66d998c467]) successfully deleted
I20250109 20:52:48.156853 989 leader_election.cc:304] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f
I20250109 20:52:48.157819 1451 raft_consensus.cc:2798] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:48.157930 1451 raft_consensus.cc:695] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:48.158177 1451 consensus_queue.cc:237] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.157534 1176 tablet_bootstrap.cc:492] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.159452 1176 ts_tablet_manager.cc:1397] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20250109 20:52:48.160324 1176 raft_consensus.cc:357] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.160499 1176 raft_consensus.cc:383] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.160584 1176 raft_consensus.cc:738] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.157610 1166 rpcz_store.cc:276] 0109 20:52:46.877749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.878321 (+ 572us) service_pool.cc:225] Handling call
0109 20:52:48.062820 (+1184499us) ts_tablet_manager.cc:1145] Deleting tablet 03b47ee787ca4dc5a4a56ae582d15b8b
0109 20:52:48.062833 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.079989 (+ 17156us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:48.135502 (+ 55513us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:48.156642 (+ 21140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1170848}
I20250109 20:52:48.160782 1176 consensus_queue.cc:260] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.161047 1166 tablet_replica.cc:331] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.161303 1176 ts_tablet_manager.cc:1428] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:48.161545 1176 tablet_bootstrap.cc:492] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.163393 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 03b47ee787ca4dc5a4a56ae582d15b8b (table default.Table_51da7d20722144bf824bdc0388d68808 [id=92af12dca25846f694464bd77a74c1e3]) successfully deleted
I20250109 20:52:48.163904 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.164314 742 tablet_service.cc:1514] Processing DeleteTablet for tablet dae574bf93d649099252e9e7b09b748c with delete_type TABLET_DATA_DELETED (replaced by tablet dc235614171f4a61b75492ddeb3f0fe6) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:48.164495 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a0d09daa2eed41c3952fa21aa2cdd09e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0d09daa2eed41c3952fa21aa2cdd09e already in progress: creating tablet
I20250109 20:52:48.169185 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4c112bab5c5a42efab7582d83ca7eac1
I20250109 20:52:48.169435 1166 raft_consensus.cc:2238] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.172608 452 master_service.cc:946] ReplaceTablet: received request to replace tablet e784a5378f7740baa39673071b56d8c5 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:48.173290 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b31b9785ee21418ba13b426ba94cfa9b consensus state has no leader
I20250109 20:52:48.175335 1176 tablet_bootstrap.cc:654] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.176203 1166 raft_consensus.cc:2267] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.179165 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c112bab5c5a42efab7582d83ca7eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.179579 1166 ts_tablet_manager.cc:1905] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.179960 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 705a7c04f3f948e1a5e486d5ccd48ccc from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:48.180979 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2ec8f8fde48b4f38881f6209ee4f4bac" } new_table_name: "default.Table_6b76c70db0cd45aa9579e976215b1470" modify_external_catalogs: true
I20250109 20:52:48.184953 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 402280f75e51463a865c0fb521d0fd8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.185161 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40e5019321c497b814a281e0ab1a7f7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:48.185336 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet c643c1ca41e048fdab95a067f899342d with delete_type TABLET_DATA_DELETED (replaced by tablet ea5d462230734767bb010f520c86c5ab) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:48.186307 990 leader_election.cc:343] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.186467 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.186556 1166 ts_tablet_manager.cc:1918] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.186652 1166 log.cc:1198] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c40e5019321c497b814a281e0ab1a7f7
I20250109 20:52:48.187220 1166 ts_tablet_manager.cc:1939] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.187662 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 326) took 1298 ms. Trace:
I20250109 20:52:48.187813 1166 rpcz_store.cc:276] 0109 20:52:46.889011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.889610 (+ 599us) service_pool.cc:225] Handling call
0109 20:52:48.160913 (+1271303us) ts_tablet_manager.cc:1145] Deleting tablet c40e5019321c497b814a281e0ab1a7f7
0109 20:52:48.160973 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.182514 (+ 21541us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:48.186416 (+ 3902us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:48.187631 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2,"tablet-delete.queue_time_us":1228875}
I20250109 20:52:48.188557 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 328) took 1253 ms. Trace:
I20250109 20:52:48.188738 1166 rpcz_store.cc:276] 0109 20:52:46.935338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.935422 (+ 84us) service_pool.cc:225] Handling call
0109 20:52:48.188341 (+1252919us) ts_tablet_manager.cc:1145] Deleting tablet 540531463c184b9abf226d7487939e60
0109 20:52:48.188419 (+ 78us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.188509 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1252045}
W20250109 20:52:48.189250 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 540531463c184b9abf226d7487939e60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 540531463c184b9abf226d7487939e60 already in progress: creating tablet
I20250109 20:52:48.189662 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 327) took 1255 ms. Trace:
I20250109 20:52:48.189844 1166 rpcz_store.cc:276] 0109 20:52:46.934382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.934516 (+ 134us) service_pool.cc:225] Handling call
0109 20:52:48.189530 (+1255014us) ts_tablet_manager.cc:1145] Deleting tablet fbde2eecdaf4470dbc74b0c59f31643c
0109 20:52:48.189541 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.189632 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253116}
W20250109 20:52:48.190361 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet fbde2eecdaf4470dbc74b0c59f31643c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbde2eecdaf4470dbc74b0c59f31643c already in progress: creating tablet
I20250109 20:52:48.190800 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 329) took 1239 ms. Trace:
I20250109 20:52:48.190917 1166 rpcz_store.cc:276] 0109 20:52:46.951249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:46.951305 (+ 56us) service_pool.cc:225] Handling call
0109 20:52:48.190693 (+1239388us) ts_tablet_manager.cc:1145] Deleting tablet 72a455033b674e5a94fc37d0275eb313
0109 20:52:48.190704 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.190776 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1238680}
W20250109 20:52:48.191308 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 72a455033b674e5a94fc37d0275eb313 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a455033b674e5a94fc37d0275eb313 already in progress: creating tablet
I20250109 20:52:48.191653 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 335) took 1146 ms. Trace:
I20250109 20:52:48.191907 1166 rpcz_store.cc:276] 0109 20:52:47.045017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:47.045076 (+ 59us) service_pool.cc:225] Handling call
0109 20:52:48.191548 (+1146472us) ts_tablet_manager.cc:1145] Deleting tablet 59e1e8754625463ea42a92ef65a89214
0109 20:52:48.191556 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.191629 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146019}
W20250109 20:52:48.192268 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 59e1e8754625463ea42a92ef65a89214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59e1e8754625463ea42a92ef65a89214 already in progress: creating tablet
W20250109 20:52:48.192464 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62caa775ef1241f68c0323a2d228c8cd consensus state has no leader
I20250109 20:52:48.192588 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 334) took 1148 ms. Trace:
I20250109 20:52:48.192668 1166 rpcz_store.cc:276] 0109 20:52:47.044640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:47.045579 (+ 939us) service_pool.cc:225] Handling call
0109 20:52:48.192495 (+1146916us) ts_tablet_manager.cc:1145] Deleting tablet 3fa23057a60a4a3eb7755e799bc3b911
0109 20:52:48.192502 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.192563 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146671}
I20250109 20:52:48.193213 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 336) took 1147 ms. Trace:
W20250109 20:52:48.193351 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3fa23057a60a4a3eb7755e799bc3b911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa23057a60a4a3eb7755e799bc3b911 already in progress: creating tablet
W20250109 20:52:48.193672 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc235614171f4a61b75492ddeb3f0fe6 has no consensus state
I20250109 20:52:48.193557 1166 rpcz_store.cc:276] 0109 20:52:47.046207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:47.046284 (+ 77us) service_pool.cc:225] Handling call
0109 20:52:48.193092 (+1146808us) ts_tablet_manager.cc:1145] Deleting tablet 3000b819a4cc4960909b98febd400631
0109 20:52:48.193102 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:48.193182 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146603}
I20250109 20:52:48.194144 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4025cdef634d4626a33f01c362a1388a
W20250109 20:52:48.194139 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3000b819a4cc4960909b98febd400631 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3000b819a4cc4960909b98febd400631 already in progress: creating tablet
I20250109 20:52:48.195165 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025cdef634d4626a33f01c362a1388a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.195631 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c112bab5c5a42efab7582d83ca7eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:48.196035 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 73e82535780c47ca8b9cd94b013b4b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73e82535780c47ca8b9cd94b013b4b42 already in progress: creating tablet
I20250109 20:52:48.196296 1166 tablet_replica.cc:331] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.196494 1166 raft_consensus.cc:2238] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.196637 1166 raft_consensus.cc:2267] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.197172 1166 ts_tablet_manager.cc:1905] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.202123 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025cdef634d4626a33f01c362a1388a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.204231 744 tablet_service.cc:1514] Processing DeleteTablet for tablet c528a1a9ae0541ba9d4a80e38e743969 with delete_type TABLET_DATA_DELETED (replaced by tablet e716270ffcf44039b075023fd0c2d2a3) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.206286 1157 ts_tablet_manager.cc:1918] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.206403 1157 log.cc:1198] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ed4866f6007d411da5cf4832fa92ece9
I20250109 20:52:48.206796 1157 ts_tablet_manager.cc:1939] T ed4866f6007d411da5cf4832fa92ece9 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:52:48.207209 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 1c65489a602141239c99e8e8dae22ab4 (table default.Table_bd7a370b6cc2487da8595058ef89e0f5 [id=2d996a757dae433a9853245f2ceb1f4f], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c65489a602141239c99e8e8dae22ab4 consensus state has no leader
I20250109 20:52:48.208076 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ed4866f6007d411da5cf4832fa92ece9 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]) successfully deleted
W20250109 20:52:48.208339 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a0d09daa2eed41c3952fa21aa2cdd09e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0d09daa2eed41c3952fa21aa2cdd09e already in progress: creating tablet
I20250109 20:52:48.209668 1157 tablet_replica.cc:331] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:48.209868 1157 raft_consensus.cc:2238] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.210016 1157 raft_consensus.cc:2267] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:48.210316 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da2d6dda69da4a59b055106c1041ec2b consensus state has no leader
W20250109 20:52:48.210541 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b77c8d6adbf44375aff2782a83643255 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b77c8d6adbf44375aff2782a83643255 already in progress: creating tablet
W20250109 20:52:48.210750 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f1c5a6b9681d4ef6a36e308ab2916423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1c5a6b9681d4ef6a36e308ab2916423 already in progress: creating tablet
W20250109 20:52:48.210930 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a89a0b47d10414d9b6a9123af1f4ab3 already in progress: creating tablet
W20250109 20:52:48.211133 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet dae574bf93d649099252e9e7b09b748c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dae574bf93d649099252e9e7b09b748c already in progress: creating tablet
W20250109 20:52:48.211352 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 402280f75e51463a865c0fb521d0fd8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 402280f75e51463a865c0fb521d0fd8a already in progress: creating tablet
I20250109 20:52:48.214239 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 705a7c04f3f948e1a5e486d5ccd48ccc
I20250109 20:52:48.214887 455 catalog_manager.cc:6596] ReplaceTablet: tablet 705a7c04f3f948e1a5e486d5ccd48ccc deleted and replaced by tablet 829bec4f66c24180afd5271057a7d0e8
I20250109 20:52:48.215039 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc with delete_type TABLET_DATA_DELETED (replaced by tablet 829bec4f66c24180afd5271057a7d0e8) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.215752 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc with delete_type TABLET_DATA_DELETED (replaced by tablet 829bec4f66c24180afd5271057a7d0e8) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.219403 1166 ts_tablet_manager.cc:1918] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.220818 1160 raft_consensus.cc:2238] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:48.221110 1483 raft_consensus.cc:2743] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:48.221979 1160 pending_rounds.cc:62] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Trying to abort 1 pending ops.
I20250109 20:52:48.222067 1160 pending_rounds.cc:69] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523644882714624 op_type: NO_OP noop_request { }
I20250109 20:52:48.223368 1157 ts_tablet_manager.cc:1905] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.224325 893 tablet_service.cc:1514] Processing DeleteTablet for tablet dae574bf93d649099252e9e7b09b748c with delete_type TABLET_DATA_DELETED (replaced by tablet dc235614171f4a61b75492ddeb3f0fe6) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.224514 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40e5019321c497b814a281e0ab1a7f7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:48.224051 1488 log.cc:926] Time spent T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250109 20:52:48.227003 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 402280f75e51463a865c0fb521d0fd8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.227599 1160 raft_consensus.cc:2883] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:48.227738 1160 raft_consensus.cc:2267] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:48.229202 989 leader_election.cc:343] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.229385 989 leader_election.cc:304] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:48.229645 989 raft_consensus.cc:2681] T c40e5019321c497b814a281e0ab1a7f7 P 5fcc647ed479491b88e5ffacf2927a1a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:52:48.232430 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e784a5378f7740baa39673071b56d8c5
I20250109 20:52:48.232795 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.233296 1165 tablet_bootstrap.cc:492] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:48.233443 1165 ts_tablet_manager.cc:1397] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.326s user 0.039s sys 0.042s
W20250109 20:52:48.233743 1488 consensus_queue.cc:1159] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "44d473955919435cb679edd5f8afe2c2" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250109 20:52:48.236557 452 catalog_manager.cc:6596] ReplaceTablet: tablet e784a5378f7740baa39673071b56d8c5 deleted and replaced by tablet 4814361a1e854451a0ee7469cc6d447b
I20250109 20:52:48.236768 742 tablet_service.cc:1467] Processing CreateTablet for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 (DEFAULT_TABLE table=default.Table_7ab416cb67944fa5b70140a8b8ab5e45 [id=5f7ff88269fa48ec80fe384008ba5780]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.236920 744 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 (DEFAULT_TABLE table=default.Table_1d1722af0fbc4d2faa77942c00d9d3d6 [id=22ffbc424e6d4038a66ddea1fca9842e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.237159 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.237449 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 (DEFAULT_TABLE table=default.Table_1d1722af0fbc4d2faa77942c00d9d3d6 [id=22ffbc424e6d4038a66ddea1fca9842e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.237826 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ec1d30b67674facb8b92a2c1a408073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.242548 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet e784a5378f7740baa39673071b56d8c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 4814361a1e854451a0ee7469cc6d447b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.243006 742 tablet_service.cc:1514] Processing DeleteTablet for tablet e784a5378f7740baa39673071b56d8c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 4814361a1e854451a0ee7469cc6d447b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.243561 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 225046d45c1c43b488babf5da8de75d7
I20250109 20:52:48.245003 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 225046d45c1c43b488babf5da8de75d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.245254 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ec1d30b67674facb8b92a2c1a408073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.246104 1166 log.cc:1198] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f1589b79a56148b6ab2946943de39fb2
I20250109 20:52:48.246527 1166 ts_tablet_manager.cc:1939] T f1589b79a56148b6ab2946943de39fb2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.247175 1016 tablet_service.cc:1467] Processing CreateTablet for tablet bd62a3ae64a143998bb1172b3a97c69f (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.247288 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 (DEFAULT_TABLE table=default.Table_7ab416cb67944fa5b70140a8b8ab5e45 [id=5f7ff88269fa48ec80fe384008ba5780]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.247515 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd62a3ae64a143998bb1172b3a97c69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.247759 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f1589b79a56148b6ab2946943de39fb2 (table default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]) successfully deleted
I20250109 20:52:48.248047 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.249511 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 712acab92e0f44a3a585d7d871498aa3 (DEFAULT_TABLE table=default.Table_52be79c241714f32810d3a84b9b0e36f [id=8bec14b8b4b74d9c87fe86950f5128e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.250198 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 (DEFAULT_TABLE table=default.Table_b590793190454bb09e3e7dd36515f65a [id=4a57bf26eda04187bd4b854d402f3c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.250969 1176 tablet_bootstrap.cc:492] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.251121 1176 ts_tablet_manager.cc:1397] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.003s
I20250109 20:52:48.251824 1176 raft_consensus.cc:357] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.252043 1176 raft_consensus.cc:383] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.252106 1176 raft_consensus.cc:738] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.252331 1176 consensus_queue.cc:260] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.252929 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 006475691733470f8a0468365972dfd2 (DEFAULT_TABLE table=default.Table_d5db66989ced4d6fb84352f19e087cc7 [id=d65cb2a049bb450391b2f32bb6bbeead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.253263 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4abc19297e4029b1b7c5396b49b967 (DEFAULT_TABLE table=default.Table_eff4d27a17b3425c92c3d388ee1cc73e [id=6f77cddda4de4f6083cedc70e3026b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.253708 1176 ts_tablet_manager.cc:1428] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:48.253647 1020 tablet_service.cc:1467] Processing CreateTablet for tablet a85c36689d8f4b16b85db5ad3c84cac0 (DEFAULT_TABLE table=default.Table_bc52c54bc34a44d88aab13982fc4e59b [id=2586e7a951ca4dfc84297301bedbd5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.257138 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712acab92e0f44a3a585d7d871498aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.257207 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet da2d6dda69da4a59b055106c1041ec2b (table default.Table_6b76c70db0cd45aa9579e976215b1470 [id=ce415eef7536445b99cab5e0769d359b]). Expected version 2 got 0
I20250109 20:52:48.259617 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c112bab5c5a42efab7582d83ca7eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.259891 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc with delete_type TABLET_DATA_DELETED (replaced by tablet 829bec4f66c24180afd5271057a7d0e8) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.260038 1160 ts_tablet_manager.cc:1905] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.264364 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 3911b54a040144baadfc4e0088d58e64 (DEFAULT_TABLE table=default.Table_1eee2fc76a024644bdd78f7fd1ef0e62 [id=24b47aa4d1f44087a21a07eba9320ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.264812 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3911b54a040144baadfc4e0088d58e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.265827 892 tablet_service.cc:1467] Processing CreateTablet for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 (DEFAULT_TABLE table=default.Table_7ab416cb67944fa5b70140a8b8ab5e45 [id=5f7ff88269fa48ec80fe384008ba5780]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.266363 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e784a5378f7740baa39673071b56d8c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 4814361a1e854451a0ee7469cc6d447b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.266717 888 tablet_service.cc:1467] Processing CreateTablet for tablet bd62a3ae64a143998bb1172b3a97c69f (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.267433 885 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4abc19297e4029b1b7c5396b49b967 (DEFAULT_TABLE table=default.Table_eff4d27a17b3425c92c3d388ee1cc73e [id=6f77cddda4de4f6083cedc70e3026b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.267629 887 tablet_service.cc:1467] Processing CreateTablet for tablet a85c36689d8f4b16b85db5ad3c84cac0 (DEFAULT_TABLE table=default.Table_bc52c54bc34a44d88aab13982fc4e59b [id=2586e7a951ca4dfc84297301bedbd5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.268236 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e716270ffcf44039b075023fd0c2d2a3 (DEFAULT_TABLE table=default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.268271 1165 raft_consensus.cc:357] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.268462 1165 raft_consensus.cc:383] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.268559 1165 raft_consensus.cc:738] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.268602 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e716270ffcf44039b075023fd0c2d2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.268784 1165 consensus_queue.cc:260] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.269025 1028 tablet_service.cc:1467] Processing CreateTablet for tablet da7f67026b164e0faad128d34188afb5 (DEFAULT_TABLE table=default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.269253 1165 ts_tablet_manager.cc:1428] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20250109 20:52:48.269405 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7f67026b164e0faad128d34188afb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.269521 1165 tablet_bootstrap.cc:492] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:48.269776 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 225046d45c1c43b488babf5da8de75d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.269737 897 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 (DEFAULT_TABLE table=default.Table_1d1722af0fbc4d2faa77942c00d9d3d6 [id=22ffbc424e6d4038a66ddea1fca9842e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.270180 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.270381 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4abc19297e4029b1b7c5396b49b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.270529 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025cdef634d4626a33f01c362a1388a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.273162 893 tablet_service.cc:1467] Processing CreateTablet for tablet 006475691733470f8a0468365972dfd2 (DEFAULT_TABLE table=default.Table_d5db66989ced4d6fb84352f19e087cc7 [id=d65cb2a049bb450391b2f32bb6bbeead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.273330 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ec1d30b67674facb8b92a2c1a408073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.276098 886 tablet_service.cc:1467] Processing CreateTablet for tablet 712acab92e0f44a3a585d7d871498aa3 (DEFAULT_TABLE table=default.Table_52be79c241714f32810d3a84b9b0e36f [id=8bec14b8b4b74d9c87fe86950f5128e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.278141 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd62a3ae64a143998bb1172b3a97c69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.278774 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85c36689d8f4b16b85db5ad3c84cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.279034 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006475691733470f8a0468365972dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.280817 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_94a783531318417989d76ed0dd1d8bdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:48.282588 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a5c5c4a2ddf348ac8a41f935ac5400d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:48.283854 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5c5c4a2ddf348ac8a41f935ac5400d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:48.284802 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94a783531318417989d76ed0dd1d8bdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.285290 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_69c64c9de8cd48a2b97c444a7b6e85d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:48.286723 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5aceb6f20bb841acb677692231b8f670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:48.286926 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712acab92e0f44a3a585d7d871498aa3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:48.287204 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aceb6f20bb841acb677692231b8f670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:48.289348 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69c64c9de8cd48a2b97c444a7b6e85d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.292774 895 tablet_service.cc:1467] Processing CreateTablet for tablet 3911b54a040144baadfc4e0088d58e64 (DEFAULT_TABLE table=default.Table_1eee2fc76a024644bdd78f7fd1ef0e62 [id=24b47aa4d1f44087a21a07eba9320ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.295708 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85c36689d8f4b16b85db5ad3c84cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.296940 899 tablet_service.cc:1467] Processing CreateTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 (DEFAULT_TABLE table=default.Table_b590793190454bb09e3e7dd36515f65a [id=4a57bf26eda04187bd4b854d402f3c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.298651 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3911b54a040144baadfc4e0088d58e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.299823 1165 tablet_bootstrap.cc:654] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.300504 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006475691733470f8a0468365972dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.302881 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d7587432d7345e4a225fc7d5c968c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.305498 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d7587432d7345e4a225fc7d5c968c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.305713 1176 tablet_bootstrap.cc:492] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.307412 1176 tablet_bootstrap.cc:654] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:48.311086 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 46d4cad7594346dfafdee8e4c49d74e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46d4cad7594346dfafdee8e4c49d74e6 already in progress: creating tablet
W20250109 20:52:48.311551 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6716e7428ce240fca7ca5b02f564a5df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6716e7428ce240fca7ca5b02f564a5df already in progress: creating tablet
W20250109 20:52:48.311828 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 already in progress: creating tablet
I20250109 20:52:48.312451 1158 tablet_bootstrap.cc:492] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:48.312655 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 225046d45c1c43b488babf5da8de75d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.312922 894 tablet_service.cc:1467] Processing CreateTablet for tablet da7f67026b164e0faad128d34188afb5 (DEFAULT_TABLE table=default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.313242 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7f67026b164e0faad128d34188afb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.309080 891 tablet_service.cc:1467] Processing CreateTablet for tablet e716270ffcf44039b075023fd0c2d2a3 (DEFAULT_TABLE table=default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.314198 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e716270ffcf44039b075023fd0c2d2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.314958 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4abc19297e4029b1b7c5396b49b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.316581 751 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4abc19297e4029b1b7c5396b49b967 (DEFAULT_TABLE table=default.Table_eff4d27a17b3425c92c3d388ee1cc73e [id=6f77cddda4de4f6083cedc70e3026b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.316977 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4abc19297e4029b1b7c5396b49b967. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.320786 740 tablet_service.cc:1467] Processing CreateTablet for tablet 006475691733470f8a0468365972dfd2 (DEFAULT_TABLE table=default.Table_d5db66989ced4d6fb84352f19e087cc7 [id=d65cb2a049bb450391b2f32bb6bbeead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.321162 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006475691733470f8a0468365972dfd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.323413 1268 raft_consensus.cc:491] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:48.323602 1268 raft_consensus.cc:513] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.324568 742 tablet_service.cc:1467] Processing CreateTablet for tablet bd62a3ae64a143998bb1172b3a97c69f (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.325289 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2d6dda69da4a59b055106c1041ec2b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:48.325963 1268 leader_election.cc:290] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:48.326949 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd62a3ae64a143998bb1172b3a97c69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.327145 746 tablet_service.cc:1467] Processing CreateTablet for tablet a85c36689d8f4b16b85db5ad3c84cac0 (DEFAULT_TABLE table=default.Table_bc52c54bc34a44d88aab13982fc4e59b [id=2586e7a951ca4dfc84297301bedbd5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.328289 889 tablet_service.cc:1467] Processing CreateTablet for tablet 048be84b36a1450a818f05fb36cbf3b9 (DEFAULT_TABLE table=default.Table_c01a7fe9ae3947a59c88c37c98a59cab [id=adc7cde5fa0445c0a26fafa6c692f8c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.328409 896 tablet_service.cc:1467] Processing CreateTablet for tablet 404ee892ab79427baaef6d19c57699d0 (DEFAULT_TABLE table=default.Table_3abc7e662927447b90d40c4c4ba6e04b [id=d394131690ff4ebaa3d9639758b62f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.328691 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 048be84b36a1450a818f05fb36cbf3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.328814 898 tablet_service.cc:1467] Processing CreateTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 (DEFAULT_TABLE table=default.Table_64ebceb89d2c4e8291bc096091578453 [id=94d89d19277c4e3fbf4ba420ebe0965c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.329258 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85c36689d8f4b16b85db5ad3c84cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.331902 893 tablet_service.cc:1467] Processing CreateTablet for tablet 6569b450bb7a4a78922061d2ede981fe (DEFAULT_TABLE table=default.Table_416c9e793e9d4fa69bd29e1e30365e06 [id=ee1cb175e24b4968b58f5d87eaeeefb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.332259 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.332536 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4814361a1e854451a0ee7469cc6d447b (DEFAULT_TABLE table=default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.333056 1158 ts_tablet_manager.cc:1397] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.177s user 0.006s sys 0.010s
W20250109 20:52:48.333353 712 leader_election.cc:343] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.334930 1160 ts_tablet_manager.cc:1918] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:48.338299 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 048be84b36a1450a818f05fb36cbf3b9 (DEFAULT_TABLE table=default.Table_c01a7fe9ae3947a59c88c37c98a59cab [id=adc7cde5fa0445c0a26fafa6c692f8c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.338714 1021 tablet_service.cc:1467] Processing CreateTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 (DEFAULT_TABLE table=default.Table_64ebceb89d2c4e8291bc096091578453 [id=94d89d19277c4e3fbf4ba420ebe0965c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.339355 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 404ee892ab79427baaef6d19c57699d0 (DEFAULT_TABLE table=default.Table_3abc7e662927447b90d40c4c4ba6e04b [id=d394131690ff4ebaa3d9639758b62f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.339689 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2d6dda69da4a59b055106c1041ec2b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:48.340634 1054 raft_consensus.cc:2463] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:48.341766 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 048be84b36a1450a818f05fb36cbf3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.342796 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ea5ea7fa44866b6d7160466f5a896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.343813 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404ee892ab79427baaef6d19c57699d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.344555 1020 tablet_service.cc:1467] Processing CreateTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 (DEFAULT_TABLE table=default.Table_986f5a94df1245f7930aca6efd8614a2 [id=2600eebbccdb4d098f1b48223a9edcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.344923 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce062dc0ac144a19f8fa3eef4a099b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.345302 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 4814361a1e854451a0ee7469cc6d447b (DEFAULT_TABLE table=default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.345572 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4814361a1e854451a0ee7469cc6d447b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.345691 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 6569b450bb7a4a78922061d2ede981fe (DEFAULT_TABLE table=default.Table_416c9e793e9d4fa69bd29e1e30365e06 [id=ee1cb175e24b4968b58f5d87eaeeefb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.345966 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6569b450bb7a4a78922061d2ede981fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.343963 1016 tablet_service.cc:1467] Processing CreateTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f (DEFAULT_TABLE table=default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d [id=35315fe696f042ee8700dd905bce5c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.357100 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea16fcc30f2c4b2eac53f20fcebd043f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.367367 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4870a667cd3e410590923c9e2497af54 (DEFAULT_TABLE table=default.Table_03ed60a072e941b6b5b9a8a477aa9020 [id=1e9c11e6b9294f56bb2efd10bef9af20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.368093 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4870a667cd3e410590923c9e2497af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.371799 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 (DEFAULT_TABLE table=default.Table_b590793190454bb09e3e7dd36515f65a [id=4a57bf26eda04187bd4b854d402f3c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.333074 750 tablet_service.cc:1467] Processing CreateTablet for tablet 712acab92e0f44a3a585d7d871498aa3 (DEFAULT_TABLE table=default.Table_52be79c241714f32810d3a84b9b0e36f [id=8bec14b8b4b74d9c87fe86950f5128e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.375015 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 (DEFAULT_TABLE table=default.Table_6d708581ee5c4179af45f5e8ce12bec5 [id=7a4b26558b3b43d097a1c01b0504b037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.375342 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0019a01afbcc452a9a9374c92dc131f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.384747 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 829bec4f66c24180afd5271057a7d0e8 (DEFAULT_TABLE table=default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.386576 1020 tablet_service.cc:1467] Processing CreateTablet for tablet dc235614171f4a61b75492ddeb3f0fe6 (DEFAULT_TABLE table=default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.392205 1033 tablet_service.cc:1467] Processing CreateTablet for tablet c865faed94c3493ea638826bdcd4253d (DEFAULT_TABLE table=default.Table_06d47c4610b54cb7b1b655f6dc07db72 [id=be7028b0c99c4883aded712bdd660cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.397310 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829bec4f66c24180afd5271057a7d0e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.398674 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c865faed94c3493ea638826bdcd4253d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.373596 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 2769e5906c604bd5b14df79db765ec61 (DEFAULT_TABLE table=default.Table_bea81cd569304584b538f85a10ebc45b [id=3c9451140e48432f8591841d4dfd8751]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.401187 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2769e5906c604bd5b14df79db765ec61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.410625 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc235614171f4a61b75492ddeb3f0fe6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:48.423364 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b68d23fb2df743f4afa2682f6d2e3efa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b68d23fb2df743f4afa2682f6d2e3efa already in progress: creating tablet
W20250109 20:52:48.424624 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ca816507d0c047668a51f6e7ebce18cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca816507d0c047668a51f6e7ebce18cb already in progress: creating tablet
I20250109 20:52:48.425912 1166 tablet_replica.cc:331] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.426257 1166 raft_consensus.cc:2238] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:48.427433 1166 pending_rounds.cc:62] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Trying to abort 1 pending ops.
I20250109 20:52:48.427435 1268 raft_consensus.cc:491] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:48.427712 1166 pending_rounds.cc:69] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523645578162176 op_type: NO_OP noop_request { }
I20250109 20:52:48.428758 1166 raft_consensus.cc:2883] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:48.429272 1166 raft_consensus.cc:2267] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.435639 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712acab92e0f44a3a585d7d871498aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.437361 748 tablet_service.cc:1467] Processing CreateTablet for tablet 3911b54a040144baadfc4e0088d58e64 (DEFAULT_TABLE table=default.Table_1eee2fc76a024644bdd78f7fd1ef0e62 [id=24b47aa4d1f44087a21a07eba9320ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.437651 753 tablet_service.cc:1467] Processing CreateTablet for tablet e716270ffcf44039b075023fd0c2d2a3 (DEFAULT_TABLE table=default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.437867 741 tablet_service.cc:1467] Processing CreateTablet for tablet da7f67026b164e0faad128d34188afb5 (DEFAULT_TABLE table=default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.441426 747 tablet_service.cc:1467] Processing CreateTablet for tablet 048be84b36a1450a818f05fb36cbf3b9 (DEFAULT_TABLE table=default.Table_c01a7fe9ae3947a59c88c37c98a59cab [id=adc7cde5fa0445c0a26fafa6c692f8c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.442060 713 leader_election.cc:304] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:48.442654 1166 ts_tablet_manager.cc:1905] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.442688 745 tablet_service.cc:1467] Processing CreateTablet for tablet 404ee892ab79427baaef6d19c57699d0 (DEFAULT_TABLE table=default.Table_3abc7e662927447b90d40c4c4ba6e04b [id=d394131690ff4ebaa3d9639758b62f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.448864 739 tablet_service.cc:1467] Processing CreateTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f (DEFAULT_TABLE table=default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d [id=35315fe696f042ee8700dd905bce5c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.450604 1176 tablet_bootstrap.cc:492] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.450759 1176 ts_tablet_manager.cc:1397] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.145s user 0.034s sys 0.038s
I20250109 20:52:48.450955 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d7587432d7345e4a225fc7d5c968c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.338976 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ea5ea7fa44866b6d7160466f5a896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.451601 1176 raft_consensus.cc:357] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.451802 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4814361a1e854451a0ee7469cc6d447b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.451797 1176 raft_consensus.cc:383] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.451876 1176 raft_consensus.cc:738] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.452057 1176 consensus_queue.cc:260] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.348153 882 tablet_service.cc:1467] Processing CreateTablet for tablet 2769e5906c604bd5b14df79db765ec61 (DEFAULT_TABLE table=default.Table_bea81cd569304584b538f85a10ebc45b [id=3c9451140e48432f8591841d4dfd8751]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.349072 883 tablet_service.cc:1467] Processing CreateTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 (DEFAULT_TABLE table=default.Table_986f5a94df1245f7930aca6efd8614a2 [id=2600eebbccdb4d098f1b48223a9edcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.349833 881 tablet_service.cc:1467] Processing CreateTablet for tablet 4870a667cd3e410590923c9e2497af54 (DEFAULT_TABLE table=default.Table_03ed60a072e941b6b5b9a8a477aa9020 [id=1e9c11e6b9294f56bb2efd10bef9af20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.355767 884 tablet_service.cc:1467] Processing CreateTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f (DEFAULT_TABLE table=default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d [id=35315fe696f042ee8700dd905bce5c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.356217 880 tablet_service.cc:1467] Processing CreateTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 (DEFAULT_TABLE table=default.Table_6d708581ee5c4179af45f5e8ce12bec5 [id=7a4b26558b3b43d097a1c01b0504b037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.363322 885 tablet_service.cc:1467] Processing CreateTablet for tablet dc235614171f4a61b75492ddeb3f0fe6 (DEFAULT_TABLE table=default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.456256 1176 ts_tablet_manager.cc:1428] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:48.462507 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3911b54a040144baadfc4e0088d58e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.463789 754 tablet_service.cc:1467] Processing CreateTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 (DEFAULT_TABLE table=default.Table_986f5a94df1245f7930aca6efd8614a2 [id=2600eebbccdb4d098f1b48223a9edcd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.465227 1166 ts_tablet_manager.cc:1918] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:48.465354 1166 log.cc:1198] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/1925a82eb711409ab2c5429c61edf9d8
I20250109 20:52:48.465740 1166 ts_tablet_manager.cc:1939] T 1925a82eb711409ab2c5429c61edf9d8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.459548 1176 tablet_bootstrap.cc:492] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.467137 757 tablet_service.cc:1467] Processing CreateTablet for tablet 2769e5906c604bd5b14df79db765ec61 (DEFAULT_TABLE table=default.Table_bea81cd569304584b538f85a10ebc45b [id=3c9451140e48432f8591841d4dfd8751]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.466514 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404ee892ab79427baaef6d19c57699d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.467445 758 tablet_service.cc:1467] Processing CreateTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 (DEFAULT_TABLE table=default.Table_64ebceb89d2c4e8291bc096091578453 [id=94d89d19277c4e3fbf4ba420ebe0965c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.467756 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0019a01afbcc452a9a9374c92dc131f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.469254 1176 tablet_bootstrap.cc:654] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.472950 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4870a667cd3e410590923c9e2497af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.475425 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 1925a82eb711409ab2c5429c61edf9d8 (table default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]) successfully deleted
W20250109 20:52:48.476091 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a0d09daa2eed41c3952fa21aa2cdd09e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0d09daa2eed41c3952fa21aa2cdd09e already in progress: creating tablet
I20250109 20:52:48.476846 1451 raft_consensus.cc:491] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:48.477041 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce062dc0ac144a19f8fa3eef4a099b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.476987 1451 raft_consensus.cc:513] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.477625 1451 leader_election.cc:290] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:48.478165 1166 tablet_replica.cc:331] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.478330 1166 raft_consensus.cc:2238] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.478480 1166 raft_consensus.cc:2267] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.478981 1166 ts_tablet_manager.cc:1905] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.481177 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea16fcc30f2c4b2eac53f20fcebd043f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.484555 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2769e5906c604bd5b14df79db765ec61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.495184 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6569b450bb7a4a78922061d2ede981fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.500233 1166 ts_tablet_manager.cc:1918] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.501973 1166 log.cc:1198] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c528a1a9ae0541ba9d4a80e38e743969
I20250109 20:52:48.502560 1166 ts_tablet_manager.cc:1939] T c528a1a9ae0541ba9d4a80e38e743969 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.367798 886 tablet_service.cc:1467] Processing CreateTablet for tablet c865faed94c3493ea638826bdcd4253d (DEFAULT_TABLE table=default.Table_06d47c4610b54cb7b1b655f6dc07db72 [id=be7028b0c99c4883aded712bdd660cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.407524 897 tablet_service.cc:1467] Processing CreateTablet for tablet 829bec4f66c24180afd5271057a7d0e8 (DEFAULT_TABLE table=default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.504101 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet c528a1a9ae0541ba9d4a80e38e743969 (table default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]) successfully deleted
I20250109 20:52:48.505411 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829bec4f66c24180afd5271057a7d0e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.505780 1166 tablet_replica.cc:331] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.506269 1166 raft_consensus.cc:2238] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.506923 1166 raft_consensus.cc:2267] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.507568 1166 ts_tablet_manager.cc:1905] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.519754 1166 ts_tablet_manager.cc:1918] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.521423 1166 log.cc:1198] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ed4866f6007d411da5cf4832fa92ece9
I20250109 20:52:48.521610 1160 log.cc:1198] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ed4866f6007d411da5cf4832fa92ece9
I20250109 20:52:48.521965 1166 ts_tablet_manager.cc:1939] T ed4866f6007d411da5cf4832fa92ece9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.523686 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ed4866f6007d411da5cf4832fa92ece9 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]) successfully deleted
W20250109 20:52:48.525238 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b77c8d6adbf44375aff2782a83643255 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b77c8d6adbf44375aff2782a83643255 already in progress: creating tablet
I20250109 20:52:48.525767 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ba73bbb08a4db0adfc9e72a84a22e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:48.526113 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f1c5a6b9681d4ef6a36e308ab2916423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1c5a6b9681d4ef6a36e308ab2916423 already in progress: creating tablet
I20250109 20:52:48.526633 1166 tablet_replica.cc:331] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.527038 1166 raft_consensus.cc:2238] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.527472 1166 raft_consensus.cc:2267] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.527536 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc235614171f4a61b75492ddeb3f0fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.528223 1166 ts_tablet_manager.cc:1905] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.530391 1160 ts_tablet_manager.cc:1939] T ed4866f6007d411da5cf4832fa92ece9 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:48.532493 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c865faed94c3493ea638826bdcd4253d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.539773 1166 ts_tablet_manager.cc:1918] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.541807 1166 log.cc:1198] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ff4f020c5ba643aeb11f8e08d748be46
I20250109 20:52:48.542173 1160 tablet_replica.cc:331] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:48.542357 1166 ts_tablet_manager.cc:1939] T ff4f020c5ba643aeb11f8e08d748be46 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.543741 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ff4f020c5ba643aeb11f8e08d748be46 (table default.Table_aaff98763d0f4eff96db3f5d2d85ba2a [id=7bd35efd3fad42dab4f6aa66d998c467]) successfully deleted
I20250109 20:52:48.543942 1166 tablet_replica.cc:331] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:48.544339 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a89a0b47d10414d9b6a9123af1f4ab3 already in progress: creating tablet
W20250109 20:52:48.544637 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet dae574bf93d649099252e9e7b09b748c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dae574bf93d649099252e9e7b09b748c already in progress: creating tablet
W20250109 20:52:48.544692 989 leader_election.cc:343] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.545701 1166 raft_consensus.cc:2238] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.546177 1166 raft_consensus.cc:2267] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.547343 1166 ts_tablet_manager.cc:1905] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.549327 1157 ts_tablet_manager.cc:1918] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.549662 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 048be84b36a1450a818f05fb36cbf3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.549912 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e716270ffcf44039b075023fd0c2d2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.551097 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7f67026b164e0faad128d34188afb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.463235 1268 raft_consensus.cc:513] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.556667 744 tablet_service.cc:1467] Processing CreateTablet for tablet 4870a667cd3e410590923c9e2497af54 (DEFAULT_TABLE table=default.Table_03ed60a072e941b6b5b9a8a477aa9020 [id=1e9c11e6b9294f56bb2efd10bef9af20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.552845 1160 raft_consensus.cc:2238] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.558704 1166 ts_tablet_manager.cc:1918] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.556010 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c528a1a9ae0541ba9d4a80e38e743969" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:48.560561 1166 log.cc:1198] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b5ba73bbb08a4db0adfc9e72a84a22e8
I20250109 20:52:48.561125 1166 ts_tablet_manager.cc:1939] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.562127 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b5ba73bbb08a4db0adfc9e72a84a22e8 (table default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]) successfully deleted
W20250109 20:52:48.563314 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 402280f75e51463a865c0fb521d0fd8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 402280f75e51463a865c0fb521d0fd8a already in progress: creating tablet
W20250109 20:52:48.564349 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4c112bab5c5a42efab7582d83ca7eac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c112bab5c5a42efab7582d83ca7eac1 already in progress: creating tablet
I20250109 20:52:48.564998 1166 tablet_replica.cc:331] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.565377 1166 raft_consensus.cc:2238] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.565398 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ea5ea7fa44866b6d7160466f5a896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.565860 1166 raft_consensus.cc:2267] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.567011 1166 ts_tablet_manager.cc:1905] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.570032 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ed4866f6007d411da5cf4832fa92ece9 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]) successfully deleted
W20250109 20:52:48.572738 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c528a1a9ae0541ba9d4a80e38e743969 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c528a1a9ae0541ba9d4a80e38e743969 already in progress: creating tablet
I20250109 20:52:48.573846 1268 leader_election.cc:290] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:48.574249 1158 raft_consensus.cc:357] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.574481 1158 raft_consensus.cc:383] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.574573 1158 raft_consensus.cc:738] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.574800 1158 consensus_queue.cc:260] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.575486 1158 ts_tablet_manager.cc:1428] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.242s user 0.001s sys 0.001s
I20250109 20:52:48.575829 1158 tablet_bootstrap.cc:492] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:48.577157 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ba73bbb08a4db0adfc9e72a84a22e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:48.577646 756 tablet_service.cc:1467] Processing CreateTablet for tablet 4814361a1e854451a0ee7469cc6d447b (DEFAULT_TABLE table=default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.578150 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6569b450bb7a4a78922061d2ede981fe (DEFAULT_TABLE table=default.Table_416c9e793e9d4fa69bd29e1e30365e06 [id=ee1cb175e24b4968b58f5d87eaeeefb5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:48.578226 990 leader_election.cc:343] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.578348 990 leader_election.cc:304] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:48.578533 990 raft_consensus.cc:2681] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 5fcc647ed479491b88e5ffacf2927a1a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:52:48.582669 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 (DEFAULT_TABLE table=default.Table_6d708581ee5c4179af45f5e8ce12bec5 [id=7a4b26558b3b43d097a1c01b0504b037]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:48.585203 712 leader_election.cc:343] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.585309 743 tablet_service.cc:1467] Processing CreateTablet for tablet dc235614171f4a61b75492ddeb3f0fe6 (DEFAULT_TABLE table=default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.586566 1157 log.cc:1198] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4c112bab5c5a42efab7582d83ca7eac1
I20250109 20:52:48.588318 755 tablet_service.cc:1467] Processing CreateTablet for tablet c865faed94c3493ea638826bdcd4253d (DEFAULT_TABLE table=default.Table_06d47c4610b54cb7b1b655f6dc07db72 [id=be7028b0c99c4883aded712bdd660cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.589071 1268 raft_consensus.cc:2798] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:48.589176 1268 raft_consensus.cc:491] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:48.589421 742 tablet_service.cc:1467] Processing CreateTablet for tablet 829bec4f66c24180afd5271057a7d0e8 (DEFAULT_TABLE table=default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.589980 1268 raft_consensus.cc:3054] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:48.599886 1268 raft_consensus.cc:513] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.604884 1268 leader_election.cc:290] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:48.615454 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4870a667cd3e410590923c9e2497af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.616209 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4814361a1e854451a0ee7469cc6d447b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.616389 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c865faed94c3493ea638826bdcd4253d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.616724 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404ee892ab79427baaef6d19c57699d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.617508 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6569b450bb7a4a78922061d2ede981fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.618247 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc235614171f4a61b75492ddeb3f0fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.623735 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0019a01afbcc452a9a9374c92dc131f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.624969 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829bec4f66c24180afd5271057a7d0e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.625718 1166 ts_tablet_manager.cc:1918] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.625888 1166 log.cc:1198] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c643c1ca41e048fdab95a067f899342d
I20250109 20:52:48.626339 1166 ts_tablet_manager.cc:1939] T c643c1ca41e048fdab95a067f899342d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.627352 1166 tablet_replica.cc:331] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.627425 1157 ts_tablet_manager.cc:1939] T 4c112bab5c5a42efab7582d83ca7eac1 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:48.627478 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet c643c1ca41e048fdab95a067f899342d (table default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]) successfully deleted
I20250109 20:52:48.627550 1166 raft_consensus.cc:2238] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.627692 1166 raft_consensus.cc:2267] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.628409 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4c112bab5c5a42efab7582d83ca7eac1 (table default.Table_5894addb4e7a40a8b557929948171f8b [id=b7903ec3ae3b4d0f8ef509c4c05f11f1]) successfully deleted
I20250109 20:52:48.631052 1166 ts_tablet_manager.cc:1905] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:48.633704 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4025cdef634d4626a33f01c362a1388a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4025cdef634d4626a33f01c362a1388a already in progress: creating tablet
I20250109 20:52:48.636880 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea16fcc30f2c4b2eac53f20fcebd043f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.644100 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce062dc0ac144a19f8fa3eef4a099b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.649602 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2769e5906c604bd5b14df79db765ec61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:48.650767 1157 tablet_replica.cc:331] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:48.571342 1160 raft_consensus.cc:2267] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.601305 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2d6dda69da4a59b055106c1041ec2b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:48.653985 1166 ts_tablet_manager.cc:1918] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.654129 1166 log.cc:1198] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/779153d4174b4887b5dc0bb355cb4a9d
I20250109 20:52:48.654398 1166 ts_tablet_manager.cc:1939] T 779153d4174b4887b5dc0bb355cb4a9d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.655426 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 779153d4174b4887b5dc0bb355cb4a9d (table default.Table_0f1b6adda4ab4df18b08fb9f430907e8 [id=6af19c1506c9416d979c0d938080d827]) successfully deleted
I20250109 20:52:48.655550 1166 tablet_replica.cc:331] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.655733 1166 raft_consensus.cc:2238] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.655941 1166 raft_consensus.cc:2267] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:48.656013 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4025cdef634d4626a33f01c362a1388a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4025cdef634d4626a33f01c362a1388a already in progress: creating tablet
W20250109 20:52:48.656221 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 705a7c04f3f948e1a5e486d5ccd48ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 705a7c04f3f948e1a5e486d5ccd48ccc already in progress: creating tablet
I20250109 20:52:48.656461 1166 ts_tablet_manager.cc:1905] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.661773 1160 ts_tablet_manager.cc:1905] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:48.666314 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 309488us
I20250109 20:52:48.668140 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2d6dda69da4a59b055106c1041ec2b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:48.668334 1054 raft_consensus.cc:3054] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:48.671679 1054 raft_consensus.cc:2463] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
W20250109 20:52:48.672616 713 leader_election.cc:343] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: c528a1a9ae0541ba9d4a80e38e743969
I20250109 20:52:48.673485 713 leader_election.cc:304] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:48.674073 713 leader_election.cc:304] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:48.674623 1158 tablet_bootstrap.cc:654] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.678519 1157 raft_consensus.cc:2238] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.679780 1166 ts_tablet_manager.cc:1918] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250109 20:52:48.679910 1166 log.cc:1198] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/e784a5378f7740baa39673071b56d8c5
I20250109 20:52:48.679888 1268 raft_consensus.cc:2701] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20250109 20:52:48.680876 1166 ts_tablet_manager.cc:1939] T e784a5378f7740baa39673071b56d8c5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.684891 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]) successfully deleted
W20250109 20:52:48.685204 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 225046d45c1c43b488babf5da8de75d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 225046d45c1c43b488babf5da8de75d7 already in progress: creating tablet
I20250109 20:52:48.687273 1268 raft_consensus.cc:2798] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:48.721866 1160 ts_tablet_manager.cc:1918] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:48.728948 428 kernel_stack_watchdog.cc:198] Thread 1119 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 295ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fb2b0bf0980 (unknown) at ??:0
@ 0x926cb7 __sanitizer::internal_sched_yield() at ??:?
@ 0x88c36d __interceptor_pthread_create at ??:?
@ 0x7fb2ad800db1 kudu::Thread::StartThread() at ??:0
@ 0xa5e837 kudu::Thread::Create() at ??:?
@ 0x7fb2ad8217ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fb2ad81d3bd kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fb2ad81b86f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7fb2b8f80245 kudu::consensus::Peer::SignalRequest() at ??:0
@ 0x7fb2b901375d kudu::consensus::PeerManager::SignalRequest() at ??:0
@ 0x7fb2b90566a4 kudu::consensus::RaftConsensus::Replicate() at ??:0
@ 0x7fb2ba3deeb6 kudu::tablet::OpDriver::Prepare() at ??:0
@ 0x7fb2ba3dd9da kudu::tablet::OpDriver::PrepareTask() at ??:0
@ 0x7fb2ba3e5e01 kudu::tablet::OpDriver::ExecuteAsync()::$_1::operator()() at ??:0
@ 0x7fb2ba3e5c0e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa71ecd std::function<>::operator()() at ??:?
I20250109 20:52:48.730993 1268 raft_consensus.cc:695] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:48.731361 1268 consensus_queue.cc:237] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.733088 1157 raft_consensus.cc:2267] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.747193 1160 log.cc:1198] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/1925a82eb711409ab2c5429c61edf9d8
I20250109 20:52:48.747648 1160 ts_tablet_manager.cc:1939] T 1925a82eb711409ab2c5429c61edf9d8 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:48.747615 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet da2d6dda69da4a59b055106c1041ec2b (table default.Table_6b76c70db0cd45aa9579e976215b1470 [id=ce415eef7536445b99cab5e0769d359b]). Expected version 2 got 0
I20250109 20:52:48.748739 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c4e1ecf827064d4198b94eee6e6b4120" } new_table_name: "default.Table_8ed167dde7b94790b7731c10aa30628c" modify_external_catalogs: true
I20250109 20:52:48.749280 1160 tablet_replica.cc:331] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:48.749470 1160 raft_consensus.cc:2238] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.749629 1160 raft_consensus.cc:2267] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.750104 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 1925a82eb711409ab2c5429c61edf9d8 (table default.Table_9016eedbbe184e968d3a7d60d2286789 [id=53044a76783c48ed945a98bfc0bfb90c]) successfully deleted
I20250109 20:52:48.750068 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_189b7740a7c34ecbbec972fa47180efe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:48.750236 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_64ebceb89d2c4e8291bc096091578453" } modify_external_catalogs: true
W20250109 20:52:48.750353 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b77c8d6adbf44375aff2782a83643255 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b77c8d6adbf44375aff2782a83643255 already in progress: creating tablet
I20250109 20:52:48.750391 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_64ebceb89d2c4e8291bc096091578453" } modify_external_catalogs: true
I20250109 20:52:48.750450 1160 ts_tablet_manager.cc:1905] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:48.750586 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_189b7740a7c34ecbbec972fa47180efe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.751843 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b2bd1da15a194f3ea1b0be4d86874130"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:48.752274 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2bd1da15a194f3ea1b0be4d86874130 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.752451 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_81566c8f4bd940aab3c5fca6ab8ff292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:48.752889 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81566c8f4bd940aab3c5fca6ab8ff292 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.754051 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_78b865cbf0f2431c9ee05f94caddda22" } modify_external_catalogs: true
I20250109 20:52:48.754220 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_78b865cbf0f2431c9ee05f94caddda22" } modify_external_catalogs: true
I20250109 20:52:48.755772 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e2cedbf0f0cd4d88ae098ddd22cb9972" } new_table_name: "default.Table_524731753c064cf5a3b8a7eda94add84" modify_external_catalogs: true
W20250109 20:52:48.756152 712 leader_election.cc:343] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:48.756387 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_df185d0c630644438a9e8cc5fe14446b" } modify_external_catalogs: true
I20250109 20:52:48.756534 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_df185d0c630644438a9e8cc5fe14446b" } modify_external_catalogs: true
I20250109 20:52:48.758460 1157 ts_tablet_manager.cc:1905] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.745983 1165 tablet_bootstrap.cc:492] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:48.758965 1165 ts_tablet_manager.cc:1397] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.489s user 0.003s sys 0.006s
I20250109 20:52:48.759831 1165 raft_consensus.cc:357] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.760052 1165 raft_consensus.cc:383] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.760121 1165 raft_consensus.cc:738] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.760383 1165 consensus_queue.cc:260] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.780542 1160 ts_tablet_manager.cc:1918] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.792373 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f1589b79a56148b6ab2946943de39fb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b31bd4072941d5aaf8c43dada3ccc7) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.795249 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:48.795855 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.798679 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4b61b24f5f024f8094ea0c19aeea208a" } new_table_name: "default.Table_5e6c78d0dceb4466bb81f225c45bee0d" modify_external_catalogs: true
I20250109 20:52:48.799093 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet b0fda753bbc940f7834507345f67651c with delete_type TABLET_DATA_DELETED (replaced by tablet cba68160be6145ddbbd2019c9b5f0076) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.799372 1166 tablet_replica.cc:331] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.799582 1166 raft_consensus.cc:2238] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.799759 1166 raft_consensus.cc:2267] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.800804 1166 ts_tablet_manager.cc:1905] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.806159 1166 ts_tablet_manager.cc:1918] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.806752 1166 log.cc:1198] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b0fda753bbc940f7834507345f67651c
I20250109 20:52:48.807096 1166 ts_tablet_manager.cc:1939] T b0fda753bbc940f7834507345f67651c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.808182 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b0fda753bbc940f7834507345f67651c (table default.Table_b70135c5a9b548538e791703e28af8f7 [id=f556fe2500bb47efba4056c36ca34823]) successfully deleted
I20250109 20:52:48.809170 1176 tablet_bootstrap.cc:492] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.809361 1176 ts_tablet_manager.cc:1397] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.350s user 0.076s sys 0.114s
I20250109 20:52:48.810285 1176 raft_consensus.cc:357] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.810532 1176 raft_consensus.cc:383] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.810600 1176 raft_consensus.cc:738] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.810904 1176 consensus_queue.cc:260] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.814199 1176 ts_tablet_manager.cc:1428] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250109 20:52:48.814715 1176 tablet_bootstrap.cc:492] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.817242 1176 tablet_bootstrap.cc:654] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:48.824800 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 246fd9ac96f54d008e8203e4580e7ef3 consensus state has no leader
I20250109 20:52:48.825237 1176 tablet_bootstrap.cc:492] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.825353 1176 ts_tablet_manager.cc:1397] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250109 20:52:48.825498 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet de8ea5ea7fa44866b6d7160466f5a896
I20250109 20:52:48.825989 1176 raft_consensus.cc:357] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.826532 1176 raft_consensus.cc:383] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.826622 1176 raft_consensus.cc:738] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.826831 1176 consensus_queue.cc:260] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.827033 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.827066 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_27a065584631460fa3736a814a26fad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:48.827669 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27a065584631460fa3736a814a26fad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:48.828423 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet de8ea5ea7fa44866b6d7160466f5a896 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de8ea5ea7fa44866b6d7160466f5a896 already in progress: creating tablet
I20250109 20:52:48.828940 1176 ts_tablet_manager.cc:1428] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:48.829350 1176 tablet_bootstrap.cc:492] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.831040 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fdc4d53d46874cb0a25cd71dfa7a3960" } new_table_name: "default.Table_eb7bd4795ee740f0ba9088dc52e7b383" modify_external_catalogs: true
I20250109 20:52:48.832147 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.832334 1166 tablet_replica.cc:331] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.832473 1166 raft_consensus.cc:2238] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.832610 1166 raft_consensus.cc:2267] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.833240 1166 ts_tablet_manager.cc:1905] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.832693 754 tablet_service.cc:1514] Processing DeleteTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.835311 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5e3e729549234c39a601e29da97094d4" } modify_external_catalogs: true
I20250109 20:52:48.835516 1157 ts_tablet_manager.cc:1918] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.835498 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5e3e729549234c39a601e29da97094d4" } modify_external_catalogs: true
I20250109 20:52:48.835630 1157 log.cc:1198] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/c528a1a9ae0541ba9d4a80e38e743969
I20250109 20:52:48.836203 1157 ts_tablet_manager.cc:1939] T c528a1a9ae0541ba9d4a80e38e743969 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:48.837020 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b7dc6809d6547f6b0354b64d7149bf4) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.837085 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet c528a1a9ae0541ba9d4a80e38e743969 (table default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]) successfully deleted
W20250109 20:52:48.837587 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 705a7c04f3f948e1a5e486d5ccd48ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 705a7c04f3f948e1a5e486d5ccd48ccc already in progress: creating tablet
I20250109 20:52:48.837855 1157 tablet_replica.cc:331] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:48.840487 1166 ts_tablet_manager.cc:1918] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.840571 1166 log.cc:1198] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/11d303cd42a2428fa17f29b48c4f64f5
I20250109 20:52:48.841136 1166 ts_tablet_manager.cc:1939] T 11d303cd42a2428fa17f29b48c4f64f5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.842181 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 11d303cd42a2428fa17f29b48c4f64f5 (table default.Table_5d1e5a64991f49a48a7b4286fc1ffe5f [id=a6f0a5fcb43d4509a188259adfdd0192]) successfully deleted
I20250109 20:52:48.842502 1166 tablet_replica.cc:331] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.842720 1166 raft_consensus.cc:2238] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.842873 1166 raft_consensus.cc:2267] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.843640 1166 ts_tablet_manager.cc:1905] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.844316 1157 raft_consensus.cc:2238] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:48.844638 1157 raft_consensus.cc:2267] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.846490 1176 tablet_bootstrap.cc:654] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:48.849256 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 107175us
I20250109 20:52:48.850222 1158 tablet_bootstrap.cc:492] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:48.850356 1158 ts_tablet_manager.cc:1397] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.275s user 0.003s sys 0.004s
I20250109 20:52:48.851181 1158 raft_consensus.cc:357] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.851361 1158 raft_consensus.cc:383] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.851414 1158 raft_consensus.cc:738] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.851624 1158 consensus_queue.cc:260] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.852037 1166 ts_tablet_manager.cc:1918] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.852144 1166 log.cc:1198] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/08ddedf78c0442daa9b1d05334f12b4d
I20250109 20:52:48.852815 1158 ts_tablet_manager.cc:1428] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:48.853358 1158 tablet_bootstrap.cc:492] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:48.854306 1166 ts_tablet_manager.cc:1939] T 08ddedf78c0442daa9b1d05334f12b4d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.855273 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 08ddedf78c0442daa9b1d05334f12b4d (table default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]) successfully deleted
I20250109 20:52:48.856782 1158 tablet_bootstrap.cc:654] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.857494 1176 tablet_bootstrap.cc:492] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.857635 1176 ts_tablet_manager.cc:1397] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20250109 20:52:48.857892 1529 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:41583 every 8 connection(s)
I20250109 20:52:48.858026 1157 ts_tablet_manager.cc:1905] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.858148 1296 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:41583
I20250109 20:52:48.858341 1176 raft_consensus.cc:357] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.858513 1176 raft_consensus.cc:383] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.858565 1176 raft_consensus.cc:738] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.858791 1176 consensus_queue.cc:260] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.860059 1296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
I20250109 20:52:48.860850 1176 ts_tablet_manager.cc:1428] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:48.861110 1176 tablet_bootstrap.cc:492] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.863365 1176 tablet_bootstrap.cc:654] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.863860 1157 ts_tablet_manager.cc:1918] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250109 20:52:48.863992 1157 log.cc:1198] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/e784a5378f7740baa39673071b56d8c5
I20250109 20:52:48.864488 1157 ts_tablet_manager.cc:1939] T e784a5378f7740baa39673071b56d8c5 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:48.865973 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu as pid 1296
I20250109 20:52:48.866787 1176 tablet_bootstrap.cc:492] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.866907 1176 ts_tablet_manager.cc:1397] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20250109 20:52:48.867722 1176 raft_consensus.cc:357] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.867936 1176 raft_consensus.cc:383] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.868027 1176 raft_consensus.cc:738] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.868256 1176 consensus_queue.cc:260] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.870285 1176 ts_tablet_manager.cc:1428] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:48.870625 1176 tablet_bootstrap.cc:492] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.873829 1176 tablet_bootstrap.cc:654] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.873709 451 catalog_manager.cc:5526] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:48.874074 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet da2d6dda69da4a59b055106c1041ec2b (table default.Table_6b76c70db0cd45aa9579e976215b1470 [id=ce415eef7536445b99cab5e0769d359b]). Expected version 2 got 0
I20250109 20:52:48.878921 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cd23012af5574f37944f9e26c07d05d2
I20250109 20:52:48.879594 754 tablet_service.cc:1514] Processing DeleteTablet for tablet cd23012af5574f37944f9e26c07d05d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.780701 1160 log.cc:1198] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/3fa23057a60a4a3eb7755e799bc3b911
I20250109 20:52:48.879881 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet cd23012af5574f37944f9e26c07d05d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.880154 1160 ts_tablet_manager.cc:1939] T 3fa23057a60a4a3eb7755e799bc3b911 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:52:48.880481 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cd23012af5574f37944f9e26c07d05d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd23012af5574f37944f9e26c07d05d2 already in progress: creating tablet
I20250109 20:52:48.791554 1165 ts_tablet_manager.cc:1428] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250109 20:52:48.881263 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 3fa23057a60a4a3eb7755e799bc3b911 (table default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee [id=00e9e64762824d89b0a757e997078b67]) successfully deleted
I20250109 20:52:48.800161 896 tablet_service.cc:1514] Processing DeleteTablet for tablet c528a1a9ae0541ba9d4a80e38e743969 with delete_type TABLET_DATA_DELETED (replaced by tablet e716270ffcf44039b075023fd0c2d2a3) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.881542 1165 tablet_bootstrap.cc:492] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:48.826629 897 tablet_service.cc:1514] Processing DeleteTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.883370 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a334724611e4303ba9d987033d872ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.883438 1165 tablet_bootstrap.cc:654] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.883343 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cede232c02904c21a0d46c338205fa14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:48.883599 1166 tablet_replica.cc:331] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.883814 1166 raft_consensus.cc:2238] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:48.883867 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cede232c02904c21a0d46c338205fa14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:48.883984 1166 raft_consensus.cc:2267] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.879673 893 tablet_service.cc:1514] Processing DeleteTablet for tablet cd23012af5574f37944f9e26c07d05d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.884773 1166 ts_tablet_manager.cc:1905] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.886245 1158 tablet_bootstrap.cc:492] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:48.886364 1158 ts_tablet_manager.cc:1397] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20250109 20:52:48.887066 1158 raft_consensus.cc:357] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.887264 1158 raft_consensus.cc:383] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.887331 1158 raft_consensus.cc:738] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:48.890245 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f1c5a6b9681d4ef6a36e308ab2916423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1c5a6b9681d4ef6a36e308ab2916423 already in progress: creating tablet
W20250109 20:52:48.890880 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet dae574bf93d649099252e9e7b09b748c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dae574bf93d649099252e9e7b09b748c already in progress: creating tablet
W20250109 20:52:48.891479 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 402280f75e51463a865c0fb521d0fd8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 402280f75e51463a865c0fb521d0fd8a already in progress: creating tablet
I20250109 20:52:48.891866 1166 ts_tablet_manager.cc:1918] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.891968 1166 log.cc:1198] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6a334724611e4303ba9d987033d872ba
W20250109 20:52:48.892031 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a89a0b47d10414d9b6a9123af1f4ab3 already in progress: creating tablet
I20250109 20:52:48.892267 1166 ts_tablet_manager.cc:1939] T 6a334724611e4303ba9d987033d872ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.892984 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c03524194a7b4e34a20424091e308ac4
I20250109 20:52:48.894007 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet c03524194a7b4e34a20424091e308ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.894215 893 tablet_service.cc:1514] Processing DeleteTablet for tablet c03524194a7b4e34a20424091e308ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.894279 754 tablet_service.cc:1514] Processing DeleteTablet for tablet c03524194a7b4e34a20424091e308ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:48.894655 1176 tablet_bootstrap.cc:492] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.894819 1176 ts_tablet_manager.cc:1397] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.007s
I20250109 20:52:48.895165 1160 tablet_replica.cc:331] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:48.895321 1160 raft_consensus.cc:2238] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:48.895391 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4c112bab5c5a42efab7582d83ca7eac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c112bab5c5a42efab7582d83ca7eac1 already in progress: creating tablet
I20250109 20:52:48.895550 1160 raft_consensus.cc:2267] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.895505 1176 raft_consensus.cc:357] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:48.895599 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 705a7c04f3f948e1a5e486d5ccd48ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 705a7c04f3f948e1a5e486d5ccd48ccc already in progress: creating tablet
I20250109 20:52:48.895704 1176 raft_consensus.cc:383] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.895767 1176 raft_consensus.cc:738] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.896003 1176 consensus_queue.cc:260] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.896155 1160 ts_tablet_manager.cc:1905] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.897297 1176 ts_tablet_manager.cc:1428] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:48.897542 1176 tablet_bootstrap.cc:492] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.899076 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6a334724611e4303ba9d987033d872ba (table default.Table_c3c448aaf88a40de8c2d09f88e78442f [id=b1346293729744dda93f981b02aa3124]) successfully deleted
I20250109 20:52:48.899266 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_db3d6cbe91b54958b543da192a338eb5" } new_table_name: "default.Table_bb77198aaa4844c1a0dbd08621e876ca" modify_external_catalogs: true
I20250109 20:52:48.899457 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:48.899678 1158 consensus_queue.cc:260] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:48.900017 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c03524194a7b4e34a20424091e308ac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c03524194a7b4e34a20424091e308ac4 already in progress: creating tablet
I20250109 20:52:48.900274 1166 tablet_replica.cc:331] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:48.900449 1166 raft_consensus.cc:2238] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:48.900820 1158 ts_tablet_manager.cc:1428] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250109 20:52:48.900959 1166 raft_consensus.cc:2267] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.901149 1158 tablet_bootstrap.cc:492] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:48.901793 1166 ts_tablet_manager.cc:1905] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.902662 1158 tablet_bootstrap.cc:654] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.902828 1176 tablet_bootstrap.cc:654] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:48.904278 1165 tablet_bootstrap.cc:492] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:48.904409 1165 ts_tablet_manager.cc:1397] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.001s
I20250109 20:52:48.911141 1166 ts_tablet_manager.cc:1918] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:48.911265 1166 log.cc:1198] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/8590e03e5d8e4f38aca6d4190a55771e
I20250109 20:52:48.911542 1166 ts_tablet_manager.cc:1939] T 8590e03e5d8e4f38aca6d4190a55771e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:48.912396 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 8590e03e5d8e4f38aca6d4190a55771e (table default.Table_cabbd075716c4d18aba39e549c7f0043 [id=aa04f9d0face4e1b84edffbeea2b524b]) successfully deleted
I20250109 20:52:48.916491 1176 tablet_bootstrap.cc:492] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:48.916607 1176 ts_tablet_manager.cc:1397] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.005s
I20250109 20:52:48.917222 1176 raft_consensus.cc:357] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.917213 1165 raft_consensus.cc:357] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.917383 1176 raft_consensus.cc:383] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.917392 1165 raft_consensus.cc:383] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:48.917460 1176 raft_consensus.cc:738] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.917459 1165 raft_consensus.cc:738] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:48.917687 1176 consensus_queue.cc:260] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.917677 1165 consensus_queue.cc:260] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.918985 1176 ts_tablet_manager.cc:1428] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:48.919198 1176 tablet_bootstrap.cc:492] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:48.919761 1165 ts_tablet_manager.cc:1428] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250109 20:52:48.920050 1165 tablet_bootstrap.cc:492] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:48.925647 1157 tablet_replica.cc:331] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:48.925491 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]) successfully deleted
W20250109 20:52:49.134109 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 208623us
I20250109 20:52:49.042471 1530 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:52:48.922150 1176 tablet_bootstrap.cc:654] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:49.135665 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 225046d45c1c43b488babf5da8de75d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 225046d45c1c43b488babf5da8de75d7 already in progress: creating tablet
I20250109 20:52:48.960927 449 master_service.cc:946] ReplaceTablet: received request to replace tablet e64207928ef141118e7fc8fe7bb2379b from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.096549 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.137511 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa23057a60a4a3eb7755e799bc3b911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.137784 1166 tablet_replica.cc:331] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:49.137953 1166 raft_consensus.cc:2238] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.138121 1166 raft_consensus.cc:2267] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.921593 1165 tablet_bootstrap.cc:654] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.138828 1166 ts_tablet_manager.cc:1905] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:48.971040 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e81ac284d7740eb9e743bb21d11a625" } modify_external_catalogs: true
I20250109 20:52:49.139480 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e81ac284d7740eb9e743bb21d11a625" } modify_external_catalogs: true
I20250109 20:52:48.925794 1157 raft_consensus.cc:2238] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.140951 1157 raft_consensus.cc:2267] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:48.930970 1158 tablet_bootstrap.cc:492] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:49.141234 1158 ts_tablet_manager.cc:1397] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.240s user 0.003s sys 0.000s
W20250109 20:52:49.141678 1514 raft_consensus.cc:668] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20250109 20:52:49.142009 1158 raft_consensus.cc:357] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.142236 1158 raft_consensus.cc:383] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.142222 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d with delete_type TABLET_DATA_DELETED (replaced by tablet 3d8d67b624984a0bb327d045c55476da) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.142336 1158 raft_consensus.cc:738] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.142668 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6cf525cb44c428f7a39a25c4e62dc (DEFAULT_TABLE table=default.Table_5aceb6f20bb841acb677692231b8f670 [id=a662d40e49744d56842c52a82df4df13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.142756 1157 ts_tablet_manager.cc:1905] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.142644 1158 consensus_queue.cc:260] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:48.936094 1160 ts_tablet_manager.cc:1918] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250109 20:52:49.143093 1023 tablet_service.cc:1467] Processing CreateTablet for tablet face19003d0042669e3f22553f9fe19a (DEFAULT_TABLE table=default.Table_94a783531318417989d76ed0dd1d8bdc [id=2b3bd52f32a145fdb6228cf715245ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.143205 1158 ts_tablet_manager.cc:1428] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:49.143271 1160 log.cc:1198] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/e784a5378f7740baa39673071b56d8c5
I20250109 20:52:49.143448 1033 tablet_service.cc:1467] Processing CreateTablet for tablet bdcb6780422541c89343e98e02938d0e (DEFAULT_TABLE table=default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe [id=fe5e7a3f2800480e860c07f214cfc841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.143575 1158 tablet_bootstrap.cc:492] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.143790 1160 ts_tablet_manager.cc:1939] T e784a5378f7740baa39673071b56d8c5 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.143818 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 6461f153a5994412ba05c3b687963fd1 (DEFAULT_TABLE table=default.Table_81566c8f4bd940aab3c5fca6ab8ff292 [id=37978bfd578840e4962df1552f7db45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.144169 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 1b50c91807734e86840e874886dc0749 (DEFAULT_TABLE table=default.Table_189b7740a7c34ecbbec972fa47180efe [id=902f745269374ba6a0c4ecbeeab62bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.144485 1027 tablet_service.cc:1467] Processing CreateTablet for tablet f7b4c038573a4afa879ecbcba75c35a7 (DEFAULT_TABLE table=default.Table_69c64c9de8cd48a2b97c444a7b6e85d1 [id=bf1f651775d04d3692d55110694a75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.144835 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2efcc2a6ec4b109845d8087491ca9f (DEFAULT_TABLE table=default.Table_a5c5c4a2ddf348ac8a41f935ac5400d5 [id=29268647fb064459a12b1d969b9e2c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:48.961683 897 tablet_service.cc:1514] Processing DeleteTablet for tablet b0fda753bbc940f7834507345f67651c with delete_type TABLET_DATA_DELETED (replaced by tablet cba68160be6145ddbbd2019c9b5f0076) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:48.995023 1191 raft_consensus.cc:491] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:49.145177 1021 tablet_service.cc:1467] Processing CreateTablet for tablet deedac68322e460b8d62adbe849de3f2 (DEFAULT_TABLE table=default.Table_cede232c02904c21a0d46c338205fa14 [id=3bdc7c0be0d14815a92667bde1011e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.145243 1191 raft_consensus.cc:513] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.145524 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ee208bad76734a8fa1ada438bdbbe48c (DEFAULT_TABLE table=default.Table_27a065584631460fa3736a814a26fad6 [id=238aa3a3a8e74dfba23e60c7d6f049cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.145911 1160 tablet_replica.cc:331] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.145881 754 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6cf525cb44c428f7a39a25c4e62dc (DEFAULT_TABLE table=default.Table_5aceb6f20bb841acb677692231b8f670 [id=a662d40e49744d56842c52a82df4df13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.145953 1191 leader_election.cc:290] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:49.059574 893 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6cf525cb44c428f7a39a25c4e62dc (DEFAULT_TABLE table=default.Table_5aceb6f20bb841acb677692231b8f670 [id=a662d40e49744d56842c52a82df4df13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.146121 1160 raft_consensus.cc:2238] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.146226 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a6cf525cb44c428f7a39a25c4e62dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.146273 1160 raft_consensus.cc:2267] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.146328 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a6cf525cb44c428f7a39a25c4e62dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.146799 1160 ts_tablet_manager.cc:1905] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.061431 896 tablet_service.cc:1467] Processing CreateTablet for tablet face19003d0042669e3f22553f9fe19a (DEFAULT_TABLE table=default.Table_94a783531318417989d76ed0dd1d8bdc [id=2b3bd52f32a145fdb6228cf715245ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.090174 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.149032 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet face19003d0042669e3f22553f9fe19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.149010 739 tablet_service.cc:1467] Processing CreateTablet for tablet face19003d0042669e3f22553f9fe19a (DEFAULT_TABLE table=default.Table_94a783531318417989d76ed0dd1d8bdc [id=2b3bd52f32a145fdb6228cf715245ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.149329 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet face19003d0042669e3f22553f9fe19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.096999 885 tablet_service.cc:1467] Processing CreateTablet for tablet bdcb6780422541c89343e98e02938d0e (DEFAULT_TABLE table=default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe [id=fe5e7a3f2800480e860c07f214cfc841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.152978 757 tablet_service.cc:1467] Processing CreateTablet for tablet bdcb6780422541c89343e98e02938d0e (DEFAULT_TABLE table=default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe [id=fe5e7a3f2800480e860c07f214cfc841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.153052 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcb6780422541c89343e98e02938d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.153266 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcb6780422541c89343e98e02938d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.156560 1530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Bootstrap starting.
I20250109 20:52:49.156955 744 tablet_service.cc:1467] Processing CreateTablet for tablet cc54e97e91c54afea098624bd85fcc6e (DEFAULT_TABLE table=default.Table_b2bd1da15a194f3ea1b0be4d86874130 [id=28723b313bd34951826646276c761571]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.157289 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc54e97e91c54afea098624bd85fcc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.104477 1566 tablet.cc:1721] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:49.161000 749 tablet_service.cc:1467] Processing CreateTablet for tablet 6461f153a5994412ba05c3b687963fd1 (DEFAULT_TABLE table=default.Table_81566c8f4bd940aab3c5fca6ab8ff292 [id=37978bfd578840e4962df1552f7db45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.161283 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6461f153a5994412ba05c3b687963fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.105479 1565 tablet.cc:1721] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:49.106300 886 tablet_service.cc:1467] Processing CreateTablet for tablet cc54e97e91c54afea098624bd85fcc6e (DEFAULT_TABLE table=default.Table_b2bd1da15a194f3ea1b0be4d86874130 [id=28723b313bd34951826646276c761571]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.164964 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc54e97e91c54afea098624bd85fcc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.165086 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1b50c91807734e86840e874886dc0749 (DEFAULT_TABLE table=default.Table_189b7740a7c34ecbbec972fa47180efe [id=902f745269374ba6a0c4ecbeeab62bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.165452 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b50c91807734e86840e874886dc0749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.167115 1157 ts_tablet_manager.cc:1918] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.167240 1157 log.cc:1198] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f1589b79a56148b6ab2946943de39fb2
I20250109 20:52:49.167187 758 tablet_service.cc:1467] Processing CreateTablet for tablet f7b4c038573a4afa879ecbcba75c35a7 (DEFAULT_TABLE table=default.Table_69c64c9de8cd48a2b97c444a7b6e85d1 [id=bf1f651775d04d3692d55110694a75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.167495 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b4c038573a4afa879ecbcba75c35a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.167747 1166 ts_tablet_manager.cc:1918] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.106868 895 tablet_service.cc:1467] Processing CreateTablet for tablet 6461f153a5994412ba05c3b687963fd1 (DEFAULT_TABLE table=default.Table_81566c8f4bd940aab3c5fca6ab8ff292 [id=37978bfd578840e4962df1552f7db45a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:49.168699 854 leader_election.cc:343] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: c528a1a9ae0541ba9d4a80e38e743969
I20250109 20:52:49.168861 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6461f153a5994412ba05c3b687963fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.168659 1166 log.cc:1198] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/3fa23057a60a4a3eb7755e799bc3b911
I20250109 20:52:49.169920 1157 ts_tablet_manager.cc:1939] T f1589b79a56148b6ab2946943de39fb2 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:49.170732 1166 ts_tablet_manager.cc:1939] T 3fa23057a60a4a3eb7755e799bc3b911 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:49.107398 898 tablet_service.cc:1467] Processing CreateTablet for tablet 1b50c91807734e86840e874886dc0749 (DEFAULT_TABLE table=default.Table_189b7740a7c34ecbbec972fa47180efe [id=902f745269374ba6a0c4ecbeeab62bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.172354 1160 ts_tablet_manager.cc:1918] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.172501 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b50c91807734e86840e874886dc0749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.107973 880 tablet_service.cc:1467] Processing CreateTablet for tablet f7b4c038573a4afa879ecbcba75c35a7 (DEFAULT_TABLE table=default.Table_69c64c9de8cd48a2b97c444a7b6e85d1 [id=bf1f651775d04d3692d55110694a75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.108597 884 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2efcc2a6ec4b109845d8087491ca9f (DEFAULT_TABLE table=default.Table_a5c5c4a2ddf348ac8a41f935ac5400d5 [id=29268647fb064459a12b1d969b9e2c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.109242 888 tablet_service.cc:1467] Processing CreateTablet for tablet deedac68322e460b8d62adbe849de3f2 (DEFAULT_TABLE table=default.Table_cede232c02904c21a0d46c338205fa14 [id=3bdc7c0be0d14815a92667bde1011e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.172873 1160 log.cc:1198] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/11d303cd42a2428fa17f29b48c4f64f5
I20250109 20:52:49.109889 890 tablet_service.cc:1467] Processing CreateTablet for tablet ee208bad76734a8fa1ada438bdbbe48c (DEFAULT_TABLE table=default.Table_27a065584631460fa3736a814a26fad6 [id=238aa3a3a8e74dfba23e60c7d6f049cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.173053 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2efcc2a6ec4b109845d8087491ca9f (DEFAULT_TABLE table=default.Table_a5c5c4a2ddf348ac8a41f935ac5400d5 [id=29268647fb064459a12b1d969b9e2c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.173254 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee208bad76734a8fa1ada438bdbbe48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.173372 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2efcc2a6ec4b109845d8087491ca9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.164487 1158 tablet_bootstrap.cc:654] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.177048 748 tablet_service.cc:1467] Processing CreateTablet for tablet deedac68322e460b8d62adbe849de3f2 (DEFAULT_TABLE table=default.Table_cede232c02904c21a0d46c338205fa14 [id=3bdc7c0be0d14815a92667bde1011e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.177438 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deedac68322e460b8d62adbe849de3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.179917 756 tablet_service.cc:1467] Processing CreateTablet for tablet ee208bad76734a8fa1ada438bdbbe48c (DEFAULT_TABLE table=default.Table_27a065584631460fa3736a814a26fad6 [id=238aa3a3a8e74dfba23e60c7d6f049cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.180303 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee208bad76734a8fa1ada438bdbbe48c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:48.976846 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ed7abf23ce04a268a460d8e3e97d56b consensus state has no leader
I20250109 20:52:49.183483 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b4c038573a4afa879ecbcba75c35a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.184573 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2efcc2a6ec4b109845d8087491ca9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.082913 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.112952 1568 tablet.cc:1721] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:49.185643 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deedac68322e460b8d62adbe849de3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.187251 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet e784a5378f7740baa39673071b56d8c5 (table default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]) successfully deleted
W20250109 20:52:49.187481 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 225046d45c1c43b488babf5da8de75d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 225046d45c1c43b488babf5da8de75d7 already in progress: creating tablet
W20250109 20:52:49.187700 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4025cdef634d4626a33f01c362a1388a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4025cdef634d4626a33f01c362a1388a already in progress: creating tablet
I20250109 20:52:48.994669 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.189553 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.005110 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_72bdbf7541534c67bc7d545c15638587"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.191040 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72bdbf7541534c67bc7d545c15638587 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:49.014879 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bba303032679462c8b16f706096b6d4d consensus state has no leader
I20250109 20:52:49.015547 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 2ed7abf23ce04a268a460d8e3e97d56b from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.015904 448 master_service.cc:946] ReplaceTablet: received request to replace tablet f1c5a6b9681d4ef6a36e308ab2916423 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.068696 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 06c3aa20539740cba4e5d6e852ad2909
W20250109 20:52:49.077850 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b1891cbef8f4b0cae0c15d478db4860 consensus state has no leader
I20250109 20:52:49.193383 1028 tablet_service.cc:1467] Processing CreateTablet for tablet cc54e97e91c54afea098624bd85fcc6e (DEFAULT_TABLE table=default.Table_b2bd1da15a194f3ea1b0be4d86874130 [id=28723b313bd34951826646276c761571]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.194181 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a6cf525cb44c428f7a39a25c4e62dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:49.197283 854 leader_election.cc:343] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: c528a1a9ae0541ba9d4a80e38e743969
I20250109 20:52:49.197409 854 leader_election.cc:304] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:49.198885 1166 tablet_replica.cc:331] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:49.199077 1166 raft_consensus.cc:2238] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.199141 1160 ts_tablet_manager.cc:1939] T 11d303cd42a2428fa17f29b48c4f64f5 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.199213 1166 raft_consensus.cc:2267] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.199728 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deedac68322e460b8d62adbe849de3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.199988 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b50c91807734e86840e874886dc0749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.200088 1191 raft_consensus.cc:2743] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:49.201711 1166 ts_tablet_manager.cc:1905] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.202566 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee208bad76734a8fa1ada438bdbbe48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.205106 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcb6780422541c89343e98e02938d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.209261 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6461f153a5994412ba05c3b687963fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.211320 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet face19003d0042669e3f22553f9fe19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.212842 1530 log.cc:826] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Log is configured to *not* fsync() on all Append() calls
I20250109 20:52:49.214262 1160 tablet_replica.cc:331] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.214471 1160 raft_consensus.cc:2238] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.214664 1160 raft_consensus.cc:2267] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.214881 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2efcc2a6ec4b109845d8087491ca9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.215375 1160 ts_tablet_manager.cc:1905] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.216914 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b4c038573a4afa879ecbcba75c35a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.217031 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.217485 1165 tablet_bootstrap.cc:492] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.217636 1165 ts_tablet_manager.cc:1397] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.298s user 0.007s sys 0.005s
I20250109 20:52:49.218475 1165 raft_consensus.cc:357] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.218703 1165 raft_consensus.cc:383] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.218781 1165 raft_consensus.cc:738] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.218990 1165 consensus_queue.cc:260] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.219478 1165 ts_tablet_manager.cc:1428] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:49.219779 1165 tablet_bootstrap.cc:492] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.223892 1165 tablet_bootstrap.cc:654] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.226938 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc54e97e91c54afea098624bd85fcc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.230573 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 11d303cd42a2428fa17f29b48c4f64f5 (table default.Table_5d1e5a64991f49a48a7b4286fc1ffe5f [id=a6f0a5fcb43d4509a188259adfdd0192]) successfully deleted
I20250109 20:52:49.231333 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e64207928ef141118e7fc8fe7bb2379b
I20250109 20:52:49.231904 449 catalog_manager.cc:6596] ReplaceTablet: tablet e64207928ef141118e7fc8fe7bb2379b deleted and replaced by tablet 2797e1df3d12476f89fb5155523c3935
I20250109 20:52:49.233049 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7307ebdb0c794451a6d91d23d69fddc6
I20250109 20:52:49.239993 1160 ts_tablet_manager.cc:1918] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.240108 1160 log.cc:1198] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/c528a1a9ae0541ba9d4a80e38e743969
I20250109 20:52:49.240464 1160 ts_tablet_manager.cc:1939] T c528a1a9ae0541ba9d4a80e38e743969 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.246701 1166 ts_tablet_manager.cc:1918] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.246933 1166 log.cc:1198] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/607a78e281594e9a92c6aa63ee7bdf8d
I20250109 20:52:49.247499 1166 ts_tablet_manager.cc:1939] T 607a78e281594e9a92c6aa63ee7bdf8d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:49.249035 890 tablet_service.cc:1514] Processing DeleteTablet for tablet e64207928ef141118e7fc8fe7bb2379b with delete_type TABLET_DATA_DELETED (replaced by tablet 2797e1df3d12476f89fb5155523c3935) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.249287 1176 tablet_bootstrap.cc:492] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:49.249298 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.249572 1176 ts_tablet_manager.cc:1397] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.330s user 0.019s sys 0.005s
I20250109 20:52:49.251230 1165 tablet_bootstrap.cc:492] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.251358 1165 ts_tablet_manager.cc:1397] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.006s
I20250109 20:52:49.251132 1176 raft_consensus.cc:357] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.251564 1176 raft_consensus.cc:383] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.251668 1176 raft_consensus.cc:738] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:52:49.251945 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c03524194a7b4e34a20424091e308ac4 (table default.Table_df185d0c630644438a9e8cc5fe14446b [id=d89b416c6f8d4417a6ccb26b443b2ef4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet c03524194a7b4e34a20424091e308ac4 consensus state has no leader
I20250109 20:52:49.252143 1176 consensus_queue.cc:260] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.253214 1165 raft_consensus.cc:357] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.253396 1165 raft_consensus.cc:383] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.253461 1165 raft_consensus.cc:738] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.253676 1165 consensus_queue.cc:260] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.254223 1165 ts_tablet_manager.cc:1428] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:49.261727 1176 ts_tablet_manager.cc:1428] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20250109 20:52:49.263379 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 49c6f1833ff1453f90a6ec6435248a02 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.264259 1176 tablet_bootstrap.cc:492] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:49.264482 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7f957a53ae794811bf294551bf3f6e00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.265625 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b806214625f649cfa01e6ed30915e3dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.267603 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b806214625f649cfa01e6ed30915e3dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.269846 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4f020c5ba643aeb11f8e08d748be46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.270110 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.270326 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.270519 1157 tablet_replica.cc:331] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:49.270745 1157 raft_consensus.cc:2238] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.270727 1176 tablet_bootstrap.cc:654] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.270916 1157 raft_consensus.cc:2267] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.271721 1157 ts_tablet_manager.cc:1905] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.277148 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.279484 1157 ts_tablet_manager.cc:1918] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.279654 1157 log.cc:1198] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ff4f020c5ba643aeb11f8e08d748be46
I20250109 20:52:49.280443 1165 tablet_bootstrap.cc:492] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.282091 1165 tablet_bootstrap.cc:654] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.284207 1160 tablet_replica.cc:331] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.284402 1160 raft_consensus.cc:2238] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.284554 1160 raft_consensus.cc:2267] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.288769 1176 tablet_bootstrap.cc:492] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:49.289013 1176 ts_tablet_manager.cc:1397] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.008s
I20250109 20:52:49.289588 1160 ts_tablet_manager.cc:1905] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.290865 1176 raft_consensus.cc:357] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.291177 1176 raft_consensus.cc:383] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.291287 1176 raft_consensus.cc:738] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.291636 1176 consensus_queue.cc:260] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.294155 1176 ts_tablet_manager.cc:1428] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250109 20:52:49.294602 1176 tablet_bootstrap.cc:492] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:49.299013 1176 tablet_bootstrap.cc:654] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.301729 1165 tablet_bootstrap.cc:492] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.301901 1160 ts_tablet_manager.cc:1918] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.302079 1165 ts_tablet_manager.cc:1397] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250109 20:52:49.303059 1165 raft_consensus.cc:357] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.303395 1165 raft_consensus.cc:383] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.303508 1165 raft_consensus.cc:738] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.303851 1165 consensus_queue.cc:260] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.304481 1165 ts_tablet_manager.cc:1428] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:49.304638 1160 log.cc:1198] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/b0fda753bbc940f7834507345f67651c
I20250109 20:52:49.305356 1160 ts_tablet_manager.cc:1939] T b0fda753bbc940f7834507345f67651c P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.305531 1157 ts_tablet_manager.cc:1939] T ff4f020c5ba643aeb11f8e08d748be46 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:49.305020 1165 tablet_bootstrap.cc:492] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.307209 1165 tablet_bootstrap.cc:654] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.307904 1157 tablet_replica.cc:331] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:49.308096 1157 raft_consensus.cc:2238] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.308473 1157 raft_consensus.cc:2267] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.309641 1157 ts_tablet_manager.cc:1905] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.311427 744 tablet_service.cc:1514] Processing DeleteTablet for tablet e64207928ef141118e7fc8fe7bb2379b with delete_type TABLET_DATA_DELETED (replaced by tablet 2797e1df3d12476f89fb5155523c3935) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.311681 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.312145 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet c528a1a9ae0541ba9d4a80e38e743969 (table default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]) successfully deleted
W20250109 20:52:49.312433 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet de8ea5ea7fa44866b6d7160466f5a896 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de8ea5ea7fa44866b6d7160466f5a896 already in progress: creating tablet
I20250109 20:52:49.312798 1160 tablet_replica.cc:331] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.312978 1160 raft_consensus.cc:2238] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.313174 1160 raft_consensus.cc:2267] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.315356 1160 ts_tablet_manager.cc:1905] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.321688 1157 ts_tablet_manager.cc:1918] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.323845 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 3fa23057a60a4a3eb7755e799bc3b911 (table default.Table_8a0c44e7fa1a4cefb3414e42aa6646ee [id=00e9e64762824d89b0a757e997078b67]) successfully deleted
I20250109 20:52:49.324090 1165 tablet_bootstrap.cc:492] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.324970 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f420b7e623243e093d26c95276c5af9 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.325212 1165 ts_tablet_manager.cc:1397] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.002s
I20250109 20:52:49.326067 1165 raft_consensus.cc:357] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:49.326416 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f957a53ae794811bf294551bf3f6e00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.327152 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3d586cebfe304d66a30047cfce9a2e16" } new_table_name: "default.Table_f33bf5f9431146a5b111b0448a62e85d" modify_external_catalogs: true
I20250109 20:52:49.327921 1165 raft_consensus.cc:383] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.328102 1165 raft_consensus.cc:738] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.328414 1165 consensus_queue.cc:260] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:49.326732 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cd23012af5574f37944f9e26c07d05d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd23012af5574f37944f9e26c07d05d2 already in progress: creating tablet
W20250109 20:52:49.329886 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c03524194a7b4e34a20424091e308ac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c03524194a7b4e34a20424091e308ac4 already in progress: creating tablet
I20250109 20:52:49.330530 1165 ts_tablet_manager.cc:1428] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:49.333010 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_18b95666d5cc487bb285333f58689af9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.333705 1165 tablet_bootstrap.cc:492] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.336284 1165 tablet_bootstrap.cc:654] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.339036 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c213e6faecce454c879d19b0bd0e787a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.339620 1157 log.cc:1198] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/b5ba73bbb08a4db0adfc9e72a84a22e8
I20250109 20:52:49.340085 1157 ts_tablet_manager.cc:1939] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:49.340253 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_524731753c064cf5a3b8a7eda94add84 [id=69b26a1bf8564a889d1227cfad55d0e2] alter complete (version 1)
I20250109 20:52:49.341976 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3abc7e662927447b90d40c4c4ba6e04b" } modify_external_catalogs: true
I20250109 20:52:49.342381 1176 tablet_bootstrap.cc:492] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:49.342589 1176 ts_tablet_manager.cc:1397] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.017s
I20250109 20:52:49.332755 1160 ts_tablet_manager.cc:1918] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.343112 1160 log.cc:1198] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/779153d4174b4887b5dc0bb355cb4a9d
I20250109 20:52:49.343498 1160 ts_tablet_manager.cc:1939] T 779153d4174b4887b5dc0bb355cb4a9d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.343640 1176 raft_consensus.cc:357] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.344111 1176 raft_consensus.cc:383] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.344269 1176 raft_consensus.cc:738] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.344621 1176 consensus_queue.cc:260] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.346170 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 607a78e281594e9a92c6aa63ee7bdf8d (table default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]) successfully deleted
I20250109 20:52:49.347599 1176 ts_tablet_manager.cc:1428] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250109 20:52:49.348230 1176 tablet_bootstrap.cc:492] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:49.348604 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.349385 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3da7386e032a4516b7fa5ae4b2418db3" } modify_external_catalogs: true
I20250109 20:52:49.350109 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3da7386e032a4516b7fa5ae4b2418db3" } modify_external_catalogs: true
I20250109 20:52:49.351994 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet b0fda753bbc940f7834507345f67651c (table default.Table_b70135c5a9b548538e791703e28af8f7 [id=f556fe2500bb47efba4056c36ca34823]) successfully deleted
I20250109 20:52:49.352136 1165 tablet_bootstrap.cc:492] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
W20250109 20:52:49.352242 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 06c3aa20539740cba4e5d6e852ad2909 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06c3aa20539740cba4e5d6e852ad2909 already in progress: creating tablet
I20250109 20:52:49.352283 1165 ts_tablet_manager.cc:1397] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20250109 20:52:49.352984 1165 raft_consensus.cc:357] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.353179 1165 raft_consensus.cc:383] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.353241 1165 raft_consensus.cc:738] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.353335 1176 tablet_bootstrap.cc:654] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.353456 1165 consensus_queue.cc:260] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.354301 1165 ts_tablet_manager.cc:1428] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:49.354622 1165 tablet_bootstrap.cc:492] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.355422 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.356568 1165 tablet_bootstrap.cc:654] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:49.357741 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e64207928ef141118e7fc8fe7bb2379b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e64207928ef141118e7fc8fe7bb2379b already in progress: creating tablet
W20250109 20:52:49.358040 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7307ebdb0c794451a6d91d23d69fddc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7307ebdb0c794451a6d91d23d69fddc6 already in progress: creating tablet
I20250109 20:52:49.358449 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 779153d4174b4887b5dc0bb355cb4a9d (table default.Table_0f1b6adda4ab4df18b08fb9f430907e8 [id=6af19c1506c9416d979c0d938080d827]) successfully deleted
I20250109 20:52:49.359058 895 tablet_service.cc:1514] Processing DeleteTablet for tablet c40e5019321c497b814a281e0ab1a7f7 with delete_type TABLET_DATA_DELETED (replaced by tablet f4eb5d2e7e68498bbfa47285babedaa6) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.359339 1160 tablet_replica.cc:331] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.359520 1160 raft_consensus.cc:2238] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.359685 1160 raft_consensus.cc:2267] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.360445 1160 ts_tablet_manager.cc:1905] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:49.367630 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47726b58d6714cd1b46848a16a34ff37 consensus state has no leader
I20250109 20:52:49.371198 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c112bab5c5a42efab7582d83ca7eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.371538 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f1589b79a56148b6ab2946943de39fb2 (table default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]) successfully deleted
I20250109 20:52:49.372048 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3abc7e662927447b90d40c4c4ba6e04b" } modify_external_catalogs: true
I20250109 20:52:49.372650 1530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=46 overwritten=0 applied=46 ignored=0} inserts{seen=43 ignored=0} mutations{seen=40 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:52:49.372936 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.373432 1530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Bootstrap complete.
I20250109 20:52:49.373934 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b0fda753bbc940f7834507345f67651c with delete_type TABLET_DATA_DELETED (replaced by tablet cba68160be6145ddbbd2019c9b5f0076) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.374454 1157 tablet_replica.cc:331] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:49.374744 1157 raft_consensus.cc:2238] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.374974 1157 raft_consensus.cc:2267] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.375988 1157 ts_tablet_manager.cc:1905] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.376457 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet e64207928ef141118e7fc8fe7bb2379b with delete_type TABLET_DATA_DELETED (replaced by tablet 2797e1df3d12476f89fb5155523c3935) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.377444 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:49.377655 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18b95666d5cc487bb285333f58689af9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:49.379505 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c213e6faecce454c879d19b0bd0e787a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.383589 1157 ts_tablet_manager.cc:1918] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.384317 1160 ts_tablet_manager.cc:1918] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.384481 1160 log.cc:1198] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/c40e5019321c497b814a281e0ab1a7f7
I20250109 20:52:49.384914 1160 ts_tablet_manager.cc:1939] T c40e5019321c497b814a281e0ab1a7f7 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.385790 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_416c9e793e9d4fa69bd29e1e30365e06" } modify_external_catalogs: true
I20250109 20:52:49.386015 1157 log.cc:1198] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/b0fda753bbc940f7834507345f67651c
I20250109 20:52:49.386209 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_416c9e793e9d4fa69bd29e1e30365e06" } modify_external_catalogs: true
I20250109 20:52:49.386422 1157 ts_tablet_manager.cc:1939] T b0fda753bbc940f7834507345f67651c P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:49.387660 1160 tablet_replica.cc:331] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.387903 1160 raft_consensus.cc:2238] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.388111 1160 raft_consensus.cc:2267] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.389317 1160 ts_tablet_manager.cc:1905] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.389356 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet c40e5019321c497b814a281e0ab1a7f7 (table default.Table_579b6c1d61dd4e80af5ea30b4e06dc35 [id=579828ccd3744a63b1212b6de38038ce]) successfully deleted
I20250109 20:52:49.390658 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f0c5869b639f430c8192b95d4653b24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.391191 1176 tablet_bootstrap.cc:492] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:49.392804 1176 ts_tablet_manager.cc:1397] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.007s
W20250109 20:52:49.394774 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0c5869b639f430c8192b95d4653b24c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.394658 1176 raft_consensus.cc:357] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.395455 1176 raft_consensus.cc:383] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.395905 895 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ba73bbb08a4db0adfc9e72a84a22e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.395843 1176 raft_consensus.cc:738] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.396767 1176 consensus_queue.cc:260] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:49.398074 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet de8ea5ea7fa44866b6d7160466f5a896 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de8ea5ea7fa44866b6d7160466f5a896 already in progress: creating tablet
W20250109 20:52:49.400944 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cd23012af5574f37944f9e26c07d05d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd23012af5574f37944f9e26c07d05d2 already in progress: creating tablet
W20250109 20:52:49.402136 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c03524194a7b4e34a20424091e308ac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c03524194a7b4e34a20424091e308ac4 already in progress: creating tablet
I20250109 20:52:49.402330 1176 ts_tablet_manager.cc:1428] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250109 20:52:49.403031 1158 tablet_bootstrap.cc:492] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:49.403082 1160 ts_tablet_manager.cc:1918] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.403194 1160 log.cc:1198] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4c112bab5c5a42efab7582d83ca7eac1
I20250109 20:52:49.403205 1158 ts_tablet_manager.cc:1397] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.260s user 0.039s sys 0.035s
I20250109 20:52:49.403506 1160 ts_tablet_manager.cc:1939] T 4c112bab5c5a42efab7582d83ca7eac1 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.403699 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ff4f020c5ba643aeb11f8e08d748be46 (table default.Table_aaff98763d0f4eff96db3f5d2d85ba2a [id=7bd35efd3fad42dab4f6aa66d998c467]) successfully deleted
I20250109 20:52:49.404112 1158 raft_consensus.cc:357] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.404342 1158 raft_consensus.cc:383] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.404451 1158 raft_consensus.cc:738] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.404742 1158 consensus_queue.cc:260] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.406117 1176 tablet_bootstrap.cc:492] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:49.404659 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4c112bab5c5a42efab7582d83ca7eac1 (table default.Table_5894addb4e7a40a8b557929948171f8b [id=b7903ec3ae3b4d0f8ef509c4c05f11f1]) successfully deleted
I20250109 20:52:49.408917 1160 tablet_replica.cc:331] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.404820 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f73279e617714f3e8b0e00b5e8773aeb" } new_table_name: "default.Table_3bc3ade0c5964f94802bd73bd9e69f21" modify_external_catalogs: true
I20250109 20:52:49.409149 1160 raft_consensus.cc:2238] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.409324 1160 raft_consensus.cc:2267] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.409363 1176 tablet_bootstrap.cc:654] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.410271 1160 ts_tablet_manager.cc:1905] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.410583 1158 ts_tablet_manager.cc:1428] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250109 20:52:49.410926 1158 tablet_bootstrap.cc:492] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.412828 1158 tablet_bootstrap.cc:654] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.410629 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet b5ba73bbb08a4db0adfc9e72a84a22e8 (table default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]) successfully deleted
W20250109 20:52:49.413314 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 06c3aa20539740cba4e5d6e852ad2909 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06c3aa20539740cba4e5d6e852ad2909 already in progress: creating tablet
I20250109 20:52:49.414870 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f1589b79a56148b6ab2946943de39fb2 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b31bd4072941d5aaf8c43dada3ccc7) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.418210 1160 ts_tablet_manager.cc:1918] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.418375 1160 log.cc:1198] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/b5ba73bbb08a4db0adfc9e72a84a22e8
I20250109 20:52:49.418824 1160 ts_tablet_manager.cc:1939] T b5ba73bbb08a4db0adfc9e72a84a22e8 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.420804 1165 tablet_bootstrap.cc:492] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.420991 1165 ts_tablet_manager.cc:1397] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.013s
I20250109 20:52:49.421823 1165 raft_consensus.cc:357] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.422062 1165 raft_consensus.cc:383] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.422205 1165 raft_consensus.cc:738] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.422544 1165 consensus_queue.cc:260] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.423027 1160 tablet_replica.cc:331] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.423254 1160 raft_consensus.cc:2238] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.423374 1165 ts_tablet_manager.cc:1428] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:49.423442 1160 raft_consensus.cc:2267] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.423771 1165 tablet_bootstrap.cc:492] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.425716 1160 ts_tablet_manager.cc:1905] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.429626 1176 tablet_bootstrap.cc:492] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:49.429808 1176 ts_tablet_manager.cc:1397] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20250109 20:52:49.430564 1176 raft_consensus.cc:357] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.430773 1176 raft_consensus.cc:383] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.430850 1176 raft_consensus.cc:738] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.431100 1176 consensus_queue.cc:260] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.434937 1160 ts_tablet_manager.cc:1918] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.435158 1176 ts_tablet_manager.cc:1428] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:49.435184 1160 log.cc:1198] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f1589b79a56148b6ab2946943de39fb2
I20250109 20:52:49.436803 1160 ts_tablet_manager.cc:1939] T f1589b79a56148b6ab2946943de39fb2 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.438040 1176 tablet_bootstrap.cc:492] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:49.440157 1165 tablet_bootstrap.cc:654] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.434120 1530 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:49.440914 1176 tablet_bootstrap.cc:654] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.441167 1530 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a29ab2002d4eb389cd11778988e1e9, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.443367 1530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 1.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:49.419917 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet b5ba73bbb08a4db0adfc9e72a84a22e8 (table default.Table_d8c45162ae564eb0b502f22e5bfe50bd [id=e088e812467844b6a8a2dbe7778c9525]) successfully deleted
W20250109 20:52:49.465159 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 45242.2us
I20250109 20:52:49.465770 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f1589b79a56148b6ab2946943de39fb2 (table default.Table_354ff4ca71d54c2293728987a8eb1f88 [id=2ab3ba8731124704889904608a2f0cd3]) successfully deleted
I20250109 20:52:49.468636 895 tablet_service.cc:1467] Processing CreateTablet for tablet 3af9563f6efd47059caac7c86c41d43f (DEFAULT_TABLE table=default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3 [id=bb7d5ce383b74f8ea9adc434fd534973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.468845 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3293f49190684b72a5c41031e621bd36 (DEFAULT_TABLE table=default.Table_72bdbf7541534c67bc7d545c15638587 [id=83feb9b334c14673b257b605634f7bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.469179 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af9563f6efd47059caac7c86c41d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.469337 898 tablet_service.cc:1467] Processing CreateTablet for tablet 2797e1df3d12476f89fb5155523c3935 (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.469698 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2797e1df3d12476f89fb5155523c3935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.472864 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3293f49190684b72a5c41031e621bd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.431071 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:49.458312 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2ed7abf23ce04a268a460d8e3e97d56b
I20250109 20:52:49.493611 451 catalog_manager.cc:6596] ReplaceTablet: tablet 2ed7abf23ce04a268a460d8e3e97d56b deleted and replaced by tablet 50039da210df47ab8476e28387c8a802
I20250109 20:52:49.458700 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 404ee892ab79427baaef6d19c57699d0
I20250109 20:52:49.461017 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8f420b7e623243e093d26c95276c5af9
I20250109 20:52:49.461431 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 85a95a31468b431fb4304cf6680016da
W20250109 20:52:49.499217 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35a39fac67894a208dd7ec3153b05e4a consensus state has no leader
I20250109 20:52:49.500167 453 catalog_manager.cc:6596] ReplaceTablet: tablet 8f420b7e623243e093d26c95276c5af9 deleted and replaced by tablet 2420b285905342b3820f9549c8f0c4ad
I20250109 20:52:49.500968 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 404ee892ab79427baaef6d19c57699d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.501389 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f420b7e623243e093d26c95276c5af9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2420b285905342b3820f9549c8f0c4ad) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:49.504619 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 404ee892ab79427baaef6d19c57699d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 404ee892ab79427baaef6d19c57699d0 already in progress: creating tablet
W20250109 20:52:49.504901 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8f420b7e623243e093d26c95276c5af9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f420b7e623243e093d26c95276c5af9 already in progress: creating tablet
I20250109 20:52:49.506625 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a95a31468b431fb4304cf6680016da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.507745 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b with delete_type TABLET_DATA_DELETED (replaced by tablet 50039da210df47ab8476e28387c8a802) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:49.507750 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 85a95a31468b431fb4304cf6680016da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85a95a31468b431fb4304cf6680016da already in progress: creating tablet
W20250109 20:52:49.508216 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 89650us
W20250109 20:52:49.508368 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2ed7abf23ce04a268a460d8e3e97d56b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ed7abf23ce04a268a460d8e3e97d56b already in progress: creating tablet
W20250109 20:52:49.508435 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e64207928ef141118e7fc8fe7bb2379b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e64207928ef141118e7fc8fe7bb2379b already in progress: creating tablet
W20250109 20:52:49.508683 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7307ebdb0c794451a6d91d23d69fddc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7307ebdb0c794451a6d91d23d69fddc6 already in progress: creating tablet
I20250109 20:52:49.508895 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet b0fda753bbc940f7834507345f67651c (table default.Table_b70135c5a9b548538e791703e28af8f7 [id=f556fe2500bb47efba4056c36ca34823]) successfully deleted
I20250109 20:52:49.510130 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 3af9563f6efd47059caac7c86c41d43f (DEFAULT_TABLE table=default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3 [id=bb7d5ce383b74f8ea9adc434fd534973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.510206 1158 tablet_bootstrap.cc:492] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:49.510404 1158 ts_tablet_manager.cc:1397] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.020s
I20250109 20:52:49.510937 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3af9563f6efd47059caac7c86c41d43f (DEFAULT_TABLE table=default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3 [id=bb7d5ce383b74f8ea9adc434fd534973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.511366 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af9563f6efd47059caac7c86c41d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.511330 1158 raft_consensus.cc:357] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.511626 1158 raft_consensus.cc:383] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.511703 1158 raft_consensus.cc:738] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.512384 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3293f49190684b72a5c41031e621bd36 (DEFAULT_TABLE table=default.Table_72bdbf7541534c67bc7d545c15638587 [id=83feb9b334c14673b257b605634f7bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.513007 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af9563f6efd47059caac7c86c41d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.513131 1191 raft_consensus.cc:491] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:49.513305 1191 raft_consensus.cc:513] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.514135 1191 leader_election.cc:290] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:49.514865 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2d6dda69da4a59b055106c1041ec2b" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:49.515439 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2d6dda69da4a59b055106c1041ec2b" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:49.515751 1054 raft_consensus.cc:2388] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d473955919435cb679edd5f8afe2c2 in current term 1: Already voted for candidate 09833ea46b7141d58c3899dace69e49f in this term.
I20250109 20:52:49.516458 854 leader_election.cc:304] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:49.516860 1191 raft_consensus.cc:3054] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:49.517292 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3293f49190684b72a5c41031e621bd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.519363 1191 raft_consensus.cc:2743] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:49.519747 1158 consensus_queue.cc:260] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.520174 758 tablet_service.cc:1467] Processing CreateTablet for tablet 3293f49190684b72a5c41031e621bd36 (DEFAULT_TABLE table=default.Table_72bdbf7541534c67bc7d545c15638587 [id=83feb9b334c14673b257b605634f7bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.523066 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3293f49190684b72a5c41031e621bd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.526566 1158 ts_tablet_manager.cc:1428] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20250109 20:52:49.530661 1586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:49.534304 1158 tablet_bootstrap.cc:492] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.535058 1586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:52:49.536212 1451 raft_consensus.cc:491] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:49.536393 1451 raft_consensus.cc:513] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.537863 1451 leader_election.cc:290] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:49.538534 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba0ffb1b394431ba30918911cdee7b7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:49.538820 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba0ffb1b394431ba30918911cdee7b7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:49.539368 989 leader_election.cc:343] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:49.539516 744 tablet_service.cc:1467] Processing CreateTablet for tablet 2797e1df3d12476f89fb5155523c3935 (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:49.539814 990 leader_election.cc:343] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:49.539907 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2797e1df3d12476f89fb5155523c3935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.539961 990 leader_election.cc:304] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:49.540572 1451 raft_consensus.cc:2743] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:49.533082 1530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:52:49.541821 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c112bab5c5a42efab7582d83ca7eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.542079 1166 tablet_replica.cc:331] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:49.542248 1166 raft_consensus.cc:2238] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.542404 1166 raft_consensus.cc:2267] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.542780 1158 tablet_bootstrap.cc:654] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.542877 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 2797e1df3d12476f89fb5155523c3935 (DEFAULT_TABLE table=default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.542932 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 404ee892ab79427baaef6d19c57699d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.543269 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2797e1df3d12476f89fb5155523c3935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.543474 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 404ee892ab79427baaef6d19c57699d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.544057 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f420b7e623243e093d26c95276c5af9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2420b285905342b3820f9549c8f0c4ad) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.544162 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f420b7e623243e093d26c95276c5af9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2420b285905342b3820f9549c8f0c4ad) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.544824 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a95a31468b431fb4304cf6680016da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.545394 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a95a31468b431fb4304cf6680016da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:49.545676 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 06c3aa20539740cba4e5d6e852ad2909 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06c3aa20539740cba4e5d6e852ad2909 already in progress: creating tablet
W20250109 20:52:49.545989 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e64207928ef141118e7fc8fe7bb2379b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e64207928ef141118e7fc8fe7bb2379b already in progress: creating tablet
I20250109 20:52:49.546020 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b with delete_type TABLET_DATA_DELETED (replaced by tablet 50039da210df47ab8476e28387c8a802) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.546223 1158 tablet_bootstrap.cc:492] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
W20250109 20:52:49.546200 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7307ebdb0c794451a6d91d23d69fddc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7307ebdb0c794451a6d91d23d69fddc6 already in progress: creating tablet
I20250109 20:52:49.546238 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b with delete_type TABLET_DATA_DELETED (replaced by tablet 50039da210df47ab8476e28387c8a802) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.546362 1158 ts_tablet_manager.cc:1397] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
W20250109 20:52:49.546779 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 404ee892ab79427baaef6d19c57699d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 404ee892ab79427baaef6d19c57699d0 already in progress: creating tablet
I20250109 20:52:49.547241 1158 raft_consensus.cc:357] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.547437 1158 raft_consensus.cc:383] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.547968 1158 raft_consensus.cc:738] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.549039 1166 ts_tablet_manager.cc:1905] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.550271 919 raft_consensus.cc:1270] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:49.550920 1514 consensus_queue.cc:1035] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:49.551841 1165 tablet_bootstrap.cc:492] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.551820 1054 raft_consensus.cc:1270] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:49.551983 1165 ts_tablet_manager.cc:1397] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.128s user 0.017s sys 0.012s
I20250109 20:52:49.552310 1514 consensus_queue.cc:1035] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:49.552706 1165 raft_consensus.cc:357] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.552919 1165 raft_consensus.cc:383] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.553007 1165 raft_consensus.cc:738] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.553234 1165 consensus_queue.cc:260] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.555622 1165 ts_tablet_manager.cc:1428] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:49.555989 1165 tablet_bootstrap.cc:492] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.551229 1158 consensus_queue.cc:260] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.556701 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_47acbd3655b34a70bdefe7ab105d58a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.557265 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bd7a370b6cc2487da8595058ef89e0f5" } new_table_name: "default.Table_4a9c362f5f8849d88cadca17da9fffe3" modify_external_catalogs: true
W20250109 20:52:49.557349 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47acbd3655b34a70bdefe7ab105d58a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.559940 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8303e354c4f54a6fa05206c788701396"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.560096 1158 ts_tablet_manager.cc:1428] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.014s user 0.001s sys 0.002s
W20250109 20:52:49.560405 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8303e354c4f54a6fa05206c788701396 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.561707 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8d84afcb90094c7fa27ee7102fb5088d" } new_table_name: "default.Table_01b9f979479d44219ae595df831ca7f8" modify_external_catalogs: true
I20250109 20:52:49.562305 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dd4f794be11b44ad971a871fdbea1094"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.562691 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd4f794be11b44ad971a871fdbea1094 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.564555 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet da2d6dda69da4a59b055106c1041ec2b (table default.Table_6b76c70db0cd45aa9579e976215b1470 [id=ce415eef7536445b99cab5e0769d359b]). Expected version 2 got 0
I20250109 20:52:49.565120 1165 tablet_bootstrap.cc:654] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.566190 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_52be79c241714f32810d3a84b9b0e36f" } new_table_name: "default.Table_549c58bff7534540ae1e16a216e2e14d" modify_external_catalogs: true
I20250109 20:52:49.566304 1158 tablet_bootstrap.cc:492] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.568647 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6cca94c4ba974861b95038c43ebf833a" } modify_external_catalogs: true
I20250109 20:52:49.568894 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6cca94c4ba974861b95038c43ebf833a" } modify_external_catalogs: true
I20250109 20:52:49.572057 1158 tablet_bootstrap.cc:654] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.572165 1166 ts_tablet_manager.cc:1918] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.572346 1166 log.cc:1198] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4c112bab5c5a42efab7582d83ca7eac1
I20250109 20:52:49.572777 1166 ts_tablet_manager.cc:1939] T 4c112bab5c5a42efab7582d83ca7eac1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:49.575026 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d52e3d34dc14e729a515dc24e0787d4" } modify_external_catalogs: true
I20250109 20:52:49.575250 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d52e3d34dc14e729a515dc24e0787d4" } modify_external_catalogs: true
I20250109 20:52:49.580547 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c4498a12cf0e4b99b80384de1e0a7bd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.581089 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4498a12cf0e4b99b80384de1e0a7bd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.581744 1568 tablet.cc:1721] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:49.582463 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8e9fb830afe541f08539f0852b9b1c30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.582909 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e9fb830afe541f08539f0852b9b1c30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.584371 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet da2d6dda69da4a59b055106c1041ec2b (table default.Table_6b76c70db0cd45aa9579e976215b1470 [id=ce415eef7536445b99cab5e0769d359b]). Expected version 2 got 0
I20250109 20:52:49.585602 895 tablet_service.cc:1514] Processing DeleteTablet for tablet c643c1ca41e048fdab95a067f899342d with delete_type TABLET_DATA_DELETED (replaced by tablet ea5d462230734767bb010f520c86c5ab) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.585795 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 2d96a569e6be4e01a26918d115cfbcf9 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.585765 1566 tablet.cc:1721] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:49.585866 1160 tablet_replica.cc:331] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.586063 1160 raft_consensus.cc:2238] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.586217 1160 raft_consensus.cc:2267] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.586313 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_90ff6b4b3aed4dcf8093176305e600e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.586731 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90ff6b4b3aed4dcf8093176305e600e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:49.588203 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 712acab92e0f44a3a585d7d871498aa3 consensus state has no leader
I20250109 20:52:49.588610 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b2bd1da15a194f3ea1b0be4d86874130" } new_table_name: "default.Table_90e645af6f5e4109a7f76ed0ae34b4ed" modify_external_catalogs: true
I20250109 20:52:49.589978 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 686ed5f7d5524c54a8debe1bc610bd25
I20250109 20:52:49.593433 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.594717 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.595454 1158 tablet_bootstrap.cc:492] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
W20250109 20:52:49.595590 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c65489a602141239c99e8e8dae22ab4 consensus state has no leader
I20250109 20:52:49.595620 1158 ts_tablet_manager.cc:1397] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.002s
I20250109 20:52:49.596321 1158 raft_consensus.cc:357] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.597745 449 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet da2d6dda69da4a59b055106c1041ec2b (table default.Table_6b76c70db0cd45aa9579e976215b1470 [id=ce415eef7536445b99cab5e0769d359b]). Expected version 2 got 0
I20250109 20:52:49.598536 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_748cfdfa67a04dc8bf878fccb596b642"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.599066 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_748cfdfa67a04dc8bf878fccb596b642 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.599797 1158 raft_consensus.cc:383] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.600121 1158 raft_consensus.cc:738] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.600126 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8de32e389cf24db59b472ed2bbabb3c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.600872 1158 consensus_queue.cc:260] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:49.601102 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8de32e389cf24db59b472ed2bbabb3c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.601769 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6569b450bb7a4a78922061d2ede981fe
I20250109 20:52:49.602243 1158 ts_tablet_manager.cc:1428] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:52:49.602566 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6569b450bb7a4a78922061d2ede981fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.603711 1158 tablet_bootstrap.cc:492] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.610921 1158 tablet_bootstrap.cc:654] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.616873 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8539cefebb844b3098a308ca580b0e92
W20250109 20:52:49.618248 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8f420b7e623243e093d26c95276c5af9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f420b7e623243e093d26c95276c5af9 already in progress: creating tablet
W20250109 20:52:49.618404 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 71411.3us
W20250109 20:52:49.618659 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 85a95a31468b431fb4304cf6680016da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85a95a31468b431fb4304cf6680016da already in progress: creating tablet
W20250109 20:52:49.618855 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2ed7abf23ce04a268a460d8e3e97d56b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ed7abf23ce04a268a460d8e3e97d56b already in progress: creating tablet
I20250109 20:52:49.620153 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8539cefebb844b3098a308ca580b0e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.620193 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4c112bab5c5a42efab7582d83ca7eac1 (table default.Table_5894addb4e7a40a8b557929948171f8b [id=b7903ec3ae3b4d0f8ef509c4c05f11f1]) successfully deleted
I20250109 20:52:49.620337 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8539cefebb844b3098a308ca580b0e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:49.620383 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 404ee892ab79427baaef6d19c57699d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 404ee892ab79427baaef6d19c57699d0 already in progress: creating tablet
W20250109 20:52:49.620555 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8f420b7e623243e093d26c95276c5af9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f420b7e623243e093d26c95276c5af9 already in progress: creating tablet
I20250109 20:52:49.620692 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d11b9148dc42bc91348e837af73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:49.620812 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 85a95a31468b431fb4304cf6680016da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85a95a31468b431fb4304cf6680016da already in progress: creating tablet
W20250109 20:52:49.621024 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2ed7abf23ce04a268a460d8e3e97d56b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ed7abf23ce04a268a460d8e3e97d56b already in progress: creating tablet
W20250109 20:52:49.621934 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8539cefebb844b3098a308ca580b0e92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8539cefebb844b3098a308ca580b0e92 already in progress: creating tablet
I20250109 20:52:49.622225 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.622469 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.622766 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet cba0ffb1b394431ba30918911cdee7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.622996 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8539cefebb844b3098a308ca580b0e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.623472 1165 tablet_bootstrap.cc:492] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.623960 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6569b450bb7a4a78922061d2ede981fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.624444 1165 ts_tablet_manager.cc:1397] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.004s
I20250109 20:52:49.625557 1165 raft_consensus.cc:357] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:49.626669 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6569b450bb7a4a78922061d2ede981fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6569b450bb7a4a78922061d2ede981fe already in progress: creating tablet
I20250109 20:52:49.627370 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6569b450bb7a4a78922061d2ede981fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.627549 1165 raft_consensus.cc:383] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.627730 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 8890a6a93e3d48dc96955f1b51e78603 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.627880 1165 raft_consensus.cc:738] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.628499 1165 consensus_queue.cc:260] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:49.630240 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc54e97e91c54afea098624bd85fcc6e consensus state has no leader
I20250109 20:52:49.633520 1176 tablet_bootstrap.cc:492] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:49.633682 1176 ts_tablet_manager.cc:1397] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.196s user 0.030s sys 0.070s
I20250109 20:52:49.634618 1176 raft_consensus.cc:357] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.634838 1165 ts_tablet_manager.cc:1428] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250109 20:52:49.634857 1176 raft_consensus.cc:383] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.634974 1176 raft_consensus.cc:738] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.635219 1176 consensus_queue.cc:260] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.635754 1176 ts_tablet_manager.cc:1428] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:49.636070 1176 tablet_bootstrap.cc:492] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:49.638039 1176 tablet_bootstrap.cc:654] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.640447 1160 ts_tablet_manager.cc:1905] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.643515 1165 tablet_bootstrap.cc:492] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.646770 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:49.646953 744 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e (DEFAULT_TABLE table=default.Table_7f957a53ae794811bf294551bf3f6e00 [id=8097eb217f1844ec8e2dc236b2f19256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.647027 895 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e (DEFAULT_TABLE table=default.Table_7f957a53ae794811bf294551bf3f6e00 [id=8097eb217f1844ec8e2dc236b2f19256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.647435 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.647446 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.648803 739 tablet_service.cc:1467] Processing CreateTablet for tablet c0860554cac64b8882e39fa62462b1aa (DEFAULT_TABLE table=default.Table_b806214625f649cfa01e6ed30915e3dc [id=f36b9d8c432d4f108296cd5576832573]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.649196 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0860554cac64b8882e39fa62462b1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.649343 1165 tablet_bootstrap.cc:654] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.653474 757 tablet_service.cc:1467] Processing CreateTablet for tablet f2c83e709a8d4e078e67af58ec51beac (DEFAULT_TABLE table=default.Table_18b95666d5cc487bb285333f58689af9 [id=4943d76200b14b51b15563815954e38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.653554 890 tablet_service.cc:1467] Processing CreateTablet for tablet f2c83e709a8d4e078e67af58ec51beac (DEFAULT_TABLE table=default.Table_18b95666d5cc487bb285333f58689af9 [id=4943d76200b14b51b15563815954e38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.653868 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c83e709a8d4e078e67af58ec51beac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.654181 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c83e709a8d4e078e67af58ec51beac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.657004 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f8eaab37b5024ccd99aa643e72cdda0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.657519 1020 tablet_service.cc:1467] Processing CreateTablet for tablet f2c83e709a8d4e078e67af58ec51beac (DEFAULT_TABLE table=default.Table_18b95666d5cc487bb285333f58689af9 [id=4943d76200b14b51b15563815954e38e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:49.657778 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8eaab37b5024ccd99aa643e72cdda0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.657934 1016 tablet_service.cc:1467] Processing CreateTablet for tablet c0860554cac64b8882e39fa62462b1aa (DEFAULT_TABLE table=default.Table_b806214625f649cfa01e6ed30915e3dc [id=f36b9d8c432d4f108296cd5576832573]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.661082 898 tablet_service.cc:1467] Processing CreateTablet for tablet c0860554cac64b8882e39fa62462b1aa (DEFAULT_TABLE table=default.Table_b806214625f649cfa01e6ed30915e3dc [id=f36b9d8c432d4f108296cd5576832573]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.661491 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0860554cac64b8882e39fa62462b1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.662901 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:49.663601 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 686ed5f7d5524c54a8debe1bc610bd25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 686ed5f7d5524c54a8debe1bc610bd25 already in progress: creating tablet
I20250109 20:52:49.664139 1165 tablet_bootstrap.cc:492] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.664992 1165 ts_tablet_manager.cc:1397] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20250109 20:52:49.665122 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_6b76c70db0cd45aa9579e976215b1470 [id=ce415eef7536445b99cab5e0769d359b] alter complete (version 2)
I20250109 20:52:49.666293 1165 raft_consensus.cc:357] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.666487 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e (DEFAULT_TABLE table=default.Table_7f957a53ae794811bf294551bf3f6e00 [id=8097eb217f1844ec8e2dc236b2f19256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.668941 1165 raft_consensus.cc:383] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.669332 1165 raft_consensus.cc:738] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.669781 1160 ts_tablet_manager.cc:1918] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.669972 1160 log.cc:1198] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/c643c1ca41e048fdab95a067f899342d
I20250109 20:52:49.670418 1160 ts_tablet_manager.cc:1939] T c643c1ca41e048fdab95a067f899342d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.671656 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet c643c1ca41e048fdab95a067f899342d (table default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]) successfully deleted
I20250109 20:52:49.672960 1166 tablet_replica.cc:331] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:49.673187 1166 raft_consensus.cc:2238] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.673370 1166 raft_consensus.cc:2267] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.674733 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2c83e709a8d4e078e67af58ec51beac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.677357 1158 tablet_bootstrap.cc:492] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:49.677490 1158 ts_tablet_manager.cc:1397] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.074s user 0.013s sys 0.004s
I20250109 20:52:49.677497 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9ef276a3879c406abf2b9f0a326cde2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.678107 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ef276a3879c406abf2b9f0a326cde2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.678310 1158 raft_consensus.cc:357] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.678493 1158 raft_consensus.cc:383] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.678594 1158 raft_consensus.cc:738] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.678750 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8890a6a93e3d48dc96955f1b51e78603
I20250109 20:52:49.678831 1158 consensus_queue.cc:260] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.679210 452 catalog_manager.cc:6596] ReplaceTablet: tablet 8890a6a93e3d48dc96955f1b51e78603 deleted and replaced by tablet beca5f1ed9d3434fb7de90cd74a6f09b
I20250109 20:52:49.679376 1158 ts_tablet_manager.cc:1428] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:49.679816 1158 tablet_bootstrap.cc:492] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.680289 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0860554cac64b8882e39fa62462b1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.682519 1158 tablet_bootstrap.cc:654] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.683205 1176 tablet_bootstrap.cc:492] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:49.683347 1176 ts_tablet_manager.cc:1397] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250109 20:52:49.684099 1176 raft_consensus.cc:357] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.688593 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 with delete_type TABLET_DATA_DELETED (replaced by tablet beca5f1ed9d3434fb7de90cd74a6f09b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.670267 1165 consensus_queue.cc:260] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.689309 1165 ts_tablet_manager.cc:1428] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20250109 20:52:49.689649 1165 tablet_bootstrap.cc:492] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.691151 1565 tablet.cc:1721] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:49.691535 1165 tablet_bootstrap.cc:654] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:49.692814 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 686ed5f7d5524c54a8debe1bc610bd25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 686ed5f7d5524c54a8debe1bc610bd25 already in progress: creating tablet
W20250109 20:52:49.693502 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6569b450bb7a4a78922061d2ede981fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6569b450bb7a4a78922061d2ede981fe already in progress: creating tablet
I20250109 20:52:49.689311 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd6ce78438841daa663b6344f062019 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:49.694252 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8539cefebb844b3098a308ca580b0e92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8539cefebb844b3098a308ca580b0e92 already in progress: creating tablet
I20250109 20:52:49.694324 1176 raft_consensus.cc:383] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.694411 1176 raft_consensus.cc:738] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.694706 1176 consensus_queue.cc:260] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.695467 1176 ts_tablet_manager.cc:1428] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20250109 20:52:49.696178 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.697275 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 with delete_type TABLET_DATA_DELETED (replaced by tablet beca5f1ed9d3434fb7de90cd74a6f09b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.697516 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 with delete_type TABLET_DATA_DELETED (replaced by tablet beca5f1ed9d3434fb7de90cd74a6f09b) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:49.698760 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8890a6a93e3d48dc96955f1b51e78603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8890a6a93e3d48dc96955f1b51e78603 already in progress: creating tablet
I20250109 20:52:49.699785 1166 ts_tablet_manager.cc:1905] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.700129 1158 tablet_bootstrap.cc:492] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:49.700316 1158 ts_tablet_manager.cc:1397] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20250109 20:52:49.701442 1158 raft_consensus.cc:357] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.702477 1158 raft_consensus.cc:383] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.702698 1158 raft_consensus.cc:738] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:49.702694 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8890a6a93e3d48dc96955f1b51e78603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8890a6a93e3d48dc96955f1b51e78603 already in progress: creating tablet
I20250109 20:52:49.703116 1158 consensus_queue.cc:260] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.704072 1158 ts_tablet_manager.cc:1428] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:52:49.704777 1158 tablet_bootstrap.cc:492] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.706857 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_18b95666d5cc487bb285333f58689af9" } new_table_name: "default.Table_a7a796c2d1d047baa518202a6b0091b7" modify_external_catalogs: true
I20250109 20:52:49.708992 1158 tablet_bootstrap.cc:654] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.708912 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_748cfdfa67a04dc8bf878fccb596b642" } modify_external_catalogs: true
I20250109 20:52:49.709151 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_748cfdfa67a04dc8bf878fccb596b642" } modify_external_catalogs: true
I20250109 20:52:49.712159 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_03ed60a072e941b6b5b9a8a477aa9020" } modify_external_catalogs: true
I20250109 20:52:49.712393 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_03ed60a072e941b6b5b9a8a477aa9020" } modify_external_catalogs: true
W20250109 20:52:49.714494 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2c83e709a8d4e078e67af58ec51beac consensus state has no leader
I20250109 20:52:49.717051 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 8a5a6dbb823e4bd3afeb157b3c5771ba from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.723052 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.729740 1176 tablet_bootstrap.cc:492] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:49.733963 1176 tablet_bootstrap.cc:654] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.734658 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.735244 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:49.736433 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:49.739116 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.739506 1157 tablet_replica.cc:331] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:49.739689 1157 raft_consensus.cc:2238] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.739840 1157 raft_consensus.cc:2267] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.740593 1157 ts_tablet_manager.cc:1905] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.740870 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet d08a3af516364157a07af7368a5f95f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.743184 898 tablet_service.cc:1467] Processing CreateTablet for tablet 527776a5d742446eb8fe61c3c855021f (DEFAULT_TABLE table=default.Table_748cfdfa67a04dc8bf878fccb596b642 [id=c82368e2d31f4654803844337d1d41a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.744791 890 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45cd5ffe7040e385d8cdfb97ed011a (DEFAULT_TABLE table=default.Table_8de32e389cf24db59b472ed2bbabb3c0 [id=4d3659cafa3246f1a84d6a4ddd44f82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.746097 895 tablet_service.cc:1467] Processing CreateTablet for tablet 74f4bdda16554d9d81de5ca5c5205f4c (DEFAULT_TABLE table=default.Table_dd4f794be11b44ad971a871fdbea1094 [id=dfdbf927d4ff4f329975148ef22d8940]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.746623 1157 ts_tablet_manager.cc:1918] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.746990 1157 log.cc:1198] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4e9cf5b77ccd4d65b023a9517efd2b0d
I20250109 20:52:49.747356 884 tablet_service.cc:1467] Processing CreateTablet for tablet f407b76780974cf7b3635fa1bf1585ec (DEFAULT_TABLE table=default.Table_8303e354c4f54a6fa05206c788701396 [id=32d713e3cdee403cb9193769815a07fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.747670 758 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45cd5ffe7040e385d8cdfb97ed011a (DEFAULT_TABLE table=default.Table_8de32e389cf24db59b472ed2bbabb3c0 [id=4d3659cafa3246f1a84d6a4ddd44f82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.747707 751 tablet_service.cc:1467] Processing CreateTablet for tablet 527776a5d742446eb8fe61c3c855021f (DEFAULT_TABLE table=default.Table_748cfdfa67a04dc8bf878fccb596b642 [id=c82368e2d31f4654803844337d1d41a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.747906 748 tablet_service.cc:1467] Processing CreateTablet for tablet f407b76780974cf7b3635fa1bf1585ec (DEFAULT_TABLE table=default.Table_8303e354c4f54a6fa05206c788701396 [id=32d713e3cdee403cb9193769815a07fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.748059 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a45cd5ffe7040e385d8cdfb97ed011a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.746850 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d303cd42a2428fa17f29b48c4f64f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.748324 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527776a5d742446eb8fe61c3c855021f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.748572 749 tablet_service.cc:1467] Processing CreateTablet for tablet 74f4bdda16554d9d81de5ca5c5205f4c (DEFAULT_TABLE table=default.Table_dd4f794be11b44ad971a871fdbea1094 [id=dfdbf927d4ff4f329975148ef22d8940]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.748955 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f4bdda16554d9d81de5ca5c5205f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.750685 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 527776a5d742446eb8fe61c3c855021f
I20250109 20:52:49.751590 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4870a667cd3e410590923c9e2497af54
I20250109 20:52:49.754434 1157 ts_tablet_manager.cc:1939] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:49.754634 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b7dc6809d6547f6b0354b64d7149bf4) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.755475 1157 tablet_replica.cc:331] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:49.755689 1157 raft_consensus.cc:2238] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.755868 1157 raft_consensus.cc:2267] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.756505 1157 ts_tablet_manager.cc:1905] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.760685 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f407b76780974cf7b3635fa1bf1585ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.762805 1157 ts_tablet_manager.cc:1918] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.762936 1157 log.cc:1198] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/11d303cd42a2428fa17f29b48c4f64f5
I20250109 20:52:49.763324 1157 ts_tablet_manager.cc:1939] T 11d303cd42a2428fa17f29b48c4f64f5 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:49.763657 1451 raft_consensus.cc:491] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:49.763867 1451 raft_consensus.cc:513] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.764761 1451 leader_election.cc:290] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:49.765465 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08a3af516364157a07af7368a5f95f8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:49.765647 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4870a667cd3e410590923c9e2497af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.765880 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 527776a5d742446eb8fe61c3c855021f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:49.766294 990 leader_election.cc:343] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:49.769894 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08a3af516364157a07af7368a5f95f8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:49.770634 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 8a45cd5ffe7040e385d8cdfb97ed011a (DEFAULT_TABLE table=default.Table_8de32e389cf24db59b472ed2bbabb3c0 [id=4d3659cafa3246f1a84d6a4ddd44f82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.771108 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a45cd5ffe7040e385d8cdfb97ed011a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.771872 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4e9cf5b77ccd4d65b023a9517efd2b0d (table default.Table_41c33f8ca7514962afbb30abb1e704da [id=873b95c3a72b4e2f9d0e89183692ac52]) successfully deleted
I20250109 20:52:49.772456 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 11d303cd42a2428fa17f29b48c4f64f5 (table default.Table_5d1e5a64991f49a48a7b4286fc1ffe5f [id=a6f0a5fcb43d4509a188259adfdd0192]) successfully deleted
W20250109 20:52:49.772881 989 leader_election.cc:343] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:49.772890 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a334724611e4303ba9d987033d872ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.773005 989 leader_election.cc:304] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:49.772962 1157 tablet_replica.cc:331] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:49.773505 1157 raft_consensus.cc:2238] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.773571 1451 raft_consensus.cc:2743] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:49.773674 1157 raft_consensus.cc:2267] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.774053 739 tablet_service.cc:1514] Processing DeleteTablet for tablet cba0ffb1b394431ba30918911cdee7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.774384 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 4870a667cd3e410590923c9e2497af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.778234 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 527776a5d742446eb8fe61c3c855021f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.778550 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 527776a5d742446eb8fe61c3c855021f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.779292 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4870a667cd3e410590923c9e2497af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.780331 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f407b76780974cf7b3635fa1bf1585ec (DEFAULT_TABLE table=default.Table_8303e354c4f54a6fa05206c788701396 [id=32d713e3cdee403cb9193769815a07fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.780679 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f407b76780974cf7b3635fa1bf1585ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.781234 888 tablet_service.cc:1467] Processing CreateTablet for tablet 3008c672bc544000b24b75ae366d9fb9 (DEFAULT_TABLE table=default.Table_47acbd3655b34a70bdefe7ab105d58a1 [id=9917ed9319044ecd84798c91f749eb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.782452 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 74f4bdda16554d9d81de5ca5c5205f4c (DEFAULT_TABLE table=default.Table_dd4f794be11b44ad971a871fdbea1094 [id=dfdbf927d4ff4f329975148ef22d8940]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.782800 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f4bdda16554d9d81de5ca5c5205f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.782841 748 tablet_service.cc:1467] Processing CreateTablet for tablet 3008c672bc544000b24b75ae366d9fb9 (DEFAULT_TABLE table=default.Table_47acbd3655b34a70bdefe7ab105d58a1 [id=9917ed9319044ecd84798c91f749eb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.783288 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3008c672bc544000b24b75ae366d9fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.787223 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 3008c672bc544000b24b75ae366d9fb9 (DEFAULT_TABLE table=default.Table_47acbd3655b34a70bdefe7ab105d58a1 [id=9917ed9319044ecd84798c91f749eb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.787739 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3008c672bc544000b24b75ae366d9fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.788978 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 527776a5d742446eb8fe61c3c855021f (DEFAULT_TABLE table=default.Table_748cfdfa67a04dc8bf878fccb596b642 [id=c82368e2d31f4654803844337d1d41a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.789357 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527776a5d742446eb8fe61c3c855021f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.794637 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e93d193814ee46d68c0d48c550964f48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.795248 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e93d193814ee46d68c0d48c550964f48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.795707 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_98b8f1400db644be881e04403ed16d10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.796169 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98b8f1400db644be881e04403ed16d10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.798705 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8c717e446858400789dd0e84b4fb60cd (DEFAULT_TABLE table=default.Table_f0c5869b639f430c8192b95d4653b24c [id=d91b6142cba745afa4358b62c31e34e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.800781 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527776a5d742446eb8fe61c3c855021f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.802490 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:49.804968 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a45cd5ffe7040e385d8cdfb97ed011a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.796550 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e75baed496db483c91ad7464f1678e0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.805485 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e75baed496db483c91ad7464f1678e0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.809108 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f4bdda16554d9d81de5ca5c5205f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.811671 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d4b677aa33534a4d8261f52878181252" } modify_external_catalogs: true
I20250109 20:52:49.811834 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d4b677aa33534a4d8261f52878181252" } modify_external_catalogs: true
I20250109 20:52:49.813961 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f407b76780974cf7b3635fa1bf1585ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.815536 1166 ts_tablet_manager.cc:1918] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.815670 1166 log.cc:1198] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/54d11b9148dc42bc91348e837af73972
I20250109 20:52:49.816036 1166 ts_tablet_manager.cc:1939] T 54d11b9148dc42bc91348e837af73972 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:49.816977 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_06d47c4610b54cb7b1b655f6dc07db72" } new_table_name: "default.Table_c07faa9c0a64412588c149655620702d" modify_external_catalogs: true
I20250109 20:52:49.817054 1166 tablet_replica.cc:331] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:49.817260 1166 raft_consensus.cc:2238] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.817425 1166 raft_consensus.cc:2267] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.818977 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3008c672bc544000b24b75ae366d9fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.823377 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c717e446858400789dd0e84b4fb60cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.823639 1157 ts_tablet_manager.cc:1905] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.827603 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a1bedae7fe314ca1bb17950aa1914398"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.827781 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_06b1c0265e874ce39301f61e327e20ff" } new_table_name: "default.Table_26f34d2971ef4d2291e2d02e53d39461" modify_external_catalogs: true
W20250109 20:52:49.828153 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1bedae7fe314ca1bb17950aa1914398 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.827626 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_df97156287614f04b00e64a3ecfb64ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.829658 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df97156287614f04b00e64a3ecfb64ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.830891 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e9297199b4064b5f889530ac2757c3ad" } modify_external_catalogs: true
I20250109 20:52:49.831068 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e9297199b4064b5f889530ac2757c3ad" } modify_external_catalogs: true
I20250109 20:52:49.832010 1568 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:49.833168 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1b9aa886edd542fc8039fd921d1048c8" } modify_external_catalogs: true
I20250109 20:52:49.833338 1566 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:49.833356 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1b9aa886edd542fc8039fd921d1048c8" } modify_external_catalogs: true
I20250109 20:52:49.834547 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e9cf5b77ccd4d65b023a9517efd2b0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:43 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.834854 1160 tablet_replica.cc:331] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.835081 1160 raft_consensus.cc:2238] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.835265 1160 raft_consensus.cc:2267] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.836236 1160 ts_tablet_manager.cc:1905] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.836555 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_261453a8462344cc9481a9b9c9165913" } new_table_name: "default.Table_ec9106622eb347769d374f0583168a54" modify_external_catalogs: true
W20250109 20:52:49.839493 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4870a667cd3e410590923c9e2497af54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4870a667cd3e410590923c9e2497af54 already in progress: creating tablet
W20250109 20:52:49.839821 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 527776a5d742446eb8fe61c3c855021f because the tablet was not found. No further retry: Not found: Tablet not found: 527776a5d742446eb8fe61c3c855021f
I20250109 20:52:49.844487 1160 ts_tablet_manager.cc:1918] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.857609 1166 ts_tablet_manager.cc:1905] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.860977 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 54d11b9148dc42bc91348e837af73972 (table default.Table_1e48be0e18b145708a9b6fa28ae21112 [id=194d1be4f3d946ec8c493ffcc5a23899]) successfully deleted
I20250109 20:52:49.848316 1565 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:49.861665 1158 tablet_bootstrap.cc:492] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:49.861812 1158 ts_tablet_manager.cc:1397] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.157s user 0.014s sys 0.006s
I20250109 20:52:49.862540 1158 raft_consensus.cc:357] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.862788 1158 raft_consensus.cc:383] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.862859 1158 raft_consensus.cc:738] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.863135 1158 consensus_queue.cc:260] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.863667 1158 ts_tablet_manager.cc:1428] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:49.863791 758 tablet_service.cc:1467] Processing CreateTablet for tablet 8c717e446858400789dd0e84b4fb60cd (DEFAULT_TABLE table=default.Table_f0c5869b639f430c8192b95d4653b24c [id=d91b6142cba745afa4358b62c31e34e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.864078 1160 log.cc:1198] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4e9cf5b77ccd4d65b023a9517efd2b0d
I20250109 20:52:49.864146 1158 tablet_bootstrap.cc:492] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.864524 1160 ts_tablet_manager.cc:1939] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.864971 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c717e446858400789dd0e84b4fb60cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.865128 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4e9cf5b77ccd4d65b023a9517efd2b0d (table default.Table_41c33f8ca7514962afbb30abb1e704da [id=873b95c3a72b4e2f9d0e89183692ac52]) successfully deleted
I20250109 20:52:49.875285 1165 tablet_bootstrap.cc:492] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.875435 1165 ts_tablet_manager.cc:1397] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.186s user 0.008s sys 0.004s
I20250109 20:52:49.875911 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 8c717e446858400789dd0e84b4fb60cd (DEFAULT_TABLE table=default.Table_f0c5869b639f430c8192b95d4653b24c [id=d91b6142cba745afa4358b62c31e34e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.876410 1165 raft_consensus.cc:357] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.876744 1165 raft_consensus.cc:383] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.876891 1165 raft_consensus.cc:738] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.877219 1165 consensus_queue.cc:260] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.877889 1165 ts_tablet_manager.cc:1428] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:49.880067 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c717e446858400789dd0e84b4fb60cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.885210 1165 tablet_bootstrap.cc:492] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.887332 1165 tablet_bootstrap.cc:654] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.887696 885 tablet_service.cc:1467] Processing CreateTablet for tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632 (DEFAULT_TABLE table=default.Table_c4498a12cf0e4b99b80384de1e0a7bd2 [id=06fb75d530e343559e65bc14300c15b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.888080 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.890389 1158 tablet_bootstrap.cc:654] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:49.897958 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c865faed94c3493ea638826bdcd4253d consensus state has no leader
I20250109 20:52:49.898236 1033 tablet_service.cc:1467] Processing CreateTablet for tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632 (DEFAULT_TABLE table=default.Table_c4498a12cf0e4b99b80384de1e0a7bd2 [id=06fb75d530e343559e65bc14300c15b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.898725 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.899533 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 2420b285905342b3820f9549c8f0c4ad (DEFAULT_TABLE table=default.Table_2e446345a181499d86750bfaca472d62 [id=ec17429d6df548118b1e53e77bac80c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.899884 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2420b285905342b3820f9549c8f0c4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.900321 757 tablet_service.cc:1467] Processing CreateTablet for tablet 9170b3a6b9bb446b8a474ae4df249409 (DEFAULT_TABLE table=default.Table_c213e6faecce454c879d19b0bd0e787a [id=7236f79dc4c34e848233f315441613d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.900929 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9170b3a6b9bb446b8a474ae4df249409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.900959 744 tablet_service.cc:1467] Processing CreateTablet for tablet 2420b285905342b3820f9549c8f0c4ad (DEFAULT_TABLE table=default.Table_2e446345a181499d86750bfaca472d62 [id=ec17429d6df548118b1e53e77bac80c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.901371 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2420b285905342b3820f9549c8f0c4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.901937 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e090f72eb1e046319b825ee5fe126332
I20250109 20:52:49.902556 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_549c58bff7534540ae1e16a216e2e14d" } new_table_name: "default.Table_3873238920c845cb8338b7a36ef75274" modify_external_catalogs: true
I20250109 20:52:49.905038 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9170b3a6b9bb446b8a474ae4df249409 (DEFAULT_TABLE table=default.Table_c213e6faecce454c879d19b0bd0e787a [id=7236f79dc4c34e848233f315441613d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.905400 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9170b3a6b9bb446b8a474ae4df249409. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:49.905898 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc32ce3dff0f4d18bb17c6784b0bab41 consensus state has no leader
I20250109 20:52:49.906248 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fca16c47746e4cbeb149a7d20fd0a979"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.906757 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fca16c47746e4cbeb149a7d20fd0a979 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.908680 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet fbe8a9e5b5fe406b988126cd1859eabc
I20250109 20:52:49.908965 898 tablet_service.cc:1514] Processing DeleteTablet for tablet e090f72eb1e046319b825ee5fe126332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.909404 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_60c85531239c48349f1931cbcf6b2977"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.909899 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60c85531239c48349f1931cbcf6b2977 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.911348 898 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.911751 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ccc647e63d744a0cbfd48bdbd78f5f87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:49.912207 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccc647e63d744a0cbfd48bdbd78f5f87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.912846 749 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.913074 739 tablet_service.cc:1514] Processing DeleteTablet for tablet e090f72eb1e046319b825ee5fe126332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.913643 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet e090f72eb1e046319b825ee5fe126332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:49.913810 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:49.914134 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9558dfe563d94482bf496e19b8a80edc consensus state has no leader
I20250109 20:52:49.914387 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 5b2fcddba2ac4d82b2d25bfc64a5967e from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:49.914598 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e090f72eb1e046319b825ee5fe126332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e090f72eb1e046319b825ee5fe126332 already in progress: creating tablet
W20250109 20:52:49.915367 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet fbe8a9e5b5fe406b988126cd1859eabc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbe8a9e5b5fe406b988126cd1859eabc already in progress: creating tablet
I20250109 20:52:49.915549 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ea477dc845614ea78e71e45f862b57ff
I20250109 20:52:49.916150 450 master_service.cc:946] ReplaceTablet: received request to replace tablet b221e048f6f6463e84e630386c304a05 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:49.916374 1157 ts_tablet_manager.cc:1918] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.916524 1157 log.cc:1198] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/08ddedf78c0442daa9b1d05334f12b4d
I20250109 20:52:49.916836 898 tablet_service.cc:1514] Processing DeleteTablet for tablet ea477dc845614ea78e71e45f862b57ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.916957 1157 ts_tablet_manager.cc:1939] T 08ddedf78c0442daa9b1d05334f12b4d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:49.917304 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ea477dc845614ea78e71e45f862b57ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:49.917946 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ea477dc845614ea78e71e45f862b57ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea477dc845614ea78e71e45f862b57ff already in progress: creating tablet
I20250109 20:52:49.918779 1158 tablet_bootstrap.cc:492] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:49.918926 1158 ts_tablet_manager.cc:1397] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20250109 20:52:49.919664 1158 raft_consensus.cc:357] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.919862 1158 raft_consensus.cc:383] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.919925 1158 raft_consensus.cc:738] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.919852 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_69a70cbdfe8148838135d7f7da4e6619"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.920153 1158 consensus_queue.cc:260] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:49.920362 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69a70cbdfe8148838135d7f7da4e6619 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.920996 1166 ts_tablet_manager.cc:1918] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.921144 1166 log.cc:1198] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4e9cf5b77ccd4d65b023a9517efd2b0d
I20250109 20:52:49.921494 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 08ddedf78c0442daa9b1d05334f12b4d (table default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]) successfully deleted
I20250109 20:52:49.921797 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e79e938de6ba4b929f53a7f9656e8416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:49.922170 1166 ts_tablet_manager.cc:1939] T 4e9cf5b77ccd4d65b023a9517efd2b0d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:49.922251 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e79e938de6ba4b929f53a7f9656e8416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:49.922415 1157 tablet_replica.cc:331] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:49.923022 1158 ts_tablet_manager.cc:1428] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:49.923262 1158 tablet_bootstrap.cc:492] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:49.923714 758 tablet_service.cc:1467] Processing CreateTablet for tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632 (DEFAULT_TABLE table=default.Table_c4498a12cf0e4b99b80384de1e0a7bd2 [id=06fb75d530e343559e65bc14300c15b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.924118 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.926390 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4e9cf5b77ccd4d65b023a9517efd2b0d (table default.Table_41c33f8ca7514962afbb30abb1e704da [id=873b95c3a72b4e2f9d0e89183692ac52]) successfully deleted
W20250109 20:52:49.926923 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 686ed5f7d5524c54a8debe1bc610bd25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 686ed5f7d5524c54a8debe1bc610bd25 already in progress: creating tablet
I20250109 20:52:49.927197 1166 tablet_replica.cc:331] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:49.927407 1166 raft_consensus.cc:2238] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.927575 1166 raft_consensus.cc:2267] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.928649 885 tablet_service.cc:1467] Processing CreateTablet for tablet 2420b285905342b3820f9549c8f0c4ad (DEFAULT_TABLE table=default.Table_2e446345a181499d86750bfaca472d62 [id=ec17429d6df548118b1e53e77bac80c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.928965 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 9170b3a6b9bb446b8a474ae4df249409 (DEFAULT_TABLE table=default.Table_c213e6faecce454c879d19b0bd0e787a [id=7236f79dc4c34e848233f315441613d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.929073 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2420b285905342b3820f9549c8f0c4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.929324 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9170b3a6b9bb446b8a474ae4df249409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.930130 1166 ts_tablet_manager.cc:1905] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.932369 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:49.932369 1157 raft_consensus.cc:2238] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.932658 1157 raft_consensus.cc:2267] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.933451 1157 ts_tablet_manager.cc:1905] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.934870 1165 tablet_bootstrap.cc:492] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:49.935083 1165 ts_tablet_manager.cc:1397] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.001s
I20250109 20:52:49.935787 1165 raft_consensus.cc:357] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.936399 1165 raft_consensus.cc:383] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.936484 1165 raft_consensus.cc:738] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.936691 1165 consensus_queue.cc:260] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.937120 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2db4e85cd4094cf0b5b699402346b980 (DEFAULT_TABLE table=default.Table_8e9fb830afe541f08539f0852b9b1c30 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.937567 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db4e85cd4094cf0b5b699402346b980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.938302 744 tablet_service.cc:1514] Processing DeleteTablet for tablet ea477dc845614ea78e71e45f862b57ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:49.945637 744 tablet_service.cc:1467] Processing CreateTablet for tablet 2db4e85cd4094cf0b5b699402346b980 (DEFAULT_TABLE table=default.Table_8e9fb830afe541f08539f0852b9b1c30 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:49.945748 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 712acab92e0f44a3a585d7d871498aa3 consensus state has no leader
I20250109 20:52:49.946113 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 2db4e85cd4094cf0b5b699402346b980 (DEFAULT_TABLE table=default.Table_8e9fb830afe541f08539f0852b9b1c30 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.946197 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7d9ef737f1e74ae1b6f66968d5c0b586" } new_table_name: "default.Table_1b4118488f5545d79852964d4b14bfe5" modify_external_catalogs: true
I20250109 20:52:49.946527 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db4e85cd4094cf0b5b699402346b980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.947430 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_01b9f979479d44219ae595df831ca7f8 [id=53718e55e4344456851f408909cc8170] alter complete (version 3)
I20250109 20:52:49.948884 1158 tablet_bootstrap.cc:654] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:49.949803 757 tablet_service.cc:1467] Processing CreateTablet for tablet 50039da210df47ab8476e28387c8a802 (DEFAULT_TABLE table=default.Table_5e6c78d0dceb4466bb81f225c45bee0d [id=b296e8173f88419a8a14ac395611b4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.950064 898 tablet_service.cc:1467] Processing CreateTablet for tablet 50039da210df47ab8476e28387c8a802 (DEFAULT_TABLE table=default.Table_5e6c78d0dceb4466bb81f225c45bee0d [id=b296e8173f88419a8a14ac395611b4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.950461 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50039da210df47ab8476e28387c8a802. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.950734 1157 ts_tablet_manager.cc:1918] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.951454 1166 ts_tablet_manager.cc:1918] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.951633 1166 log.cc:1198] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/cba0ffb1b394431ba30918911cdee7b7
I20250109 20:52:49.952052 1166 ts_tablet_manager.cc:1939] T cba0ffb1b394431ba30918911cdee7b7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:49.952390 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d11b9148dc42bc91348e837af73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.952694 895 tablet_service.cc:1514] Processing DeleteTablet for tablet cba0ffb1b394431ba30918911cdee7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:49.952859 1160 tablet_replica.cc:331] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:49.953100 1160 raft_consensus.cc:2238] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.953121 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet cba0ffb1b394431ba30918911cdee7b7 (table default.Table_609a0ae37cfd4b6d8939b4829beb84d6 [id=f06f85dff8684b1cac234aa2e35b6598]) successfully deleted
I20250109 20:52:49.953306 1160 raft_consensus.cc:2267] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:49.953718 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8539cefebb844b3098a308ca580b0e92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8539cefebb844b3098a308ca580b0e92 already in progress: creating tablet
W20250109 20:52:49.954259 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6569b450bb7a4a78922061d2ede981fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6569b450bb7a4a78922061d2ede981fe already in progress: creating tablet
I20250109 20:52:49.954326 1160 ts_tablet_manager.cc:1905] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.954584 1166 tablet_replica.cc:331] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:49.954962 1166 raft_consensus.cc:2238] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:49.955130 1157 log.cc:1198] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6a334724611e4303ba9d987033d872ba
I20250109 20:52:49.955251 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ea3137b2ba3d44e3a715f18eb25dca01" } modify_external_catalogs: true
I20250109 20:52:49.955518 1157 ts_tablet_manager.cc:1939] T 6a334724611e4303ba9d987033d872ba P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:49.955498 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ea3137b2ba3d44e3a715f18eb25dca01" } modify_external_catalogs: true
I20250109 20:52:49.955947 1166 raft_consensus.cc:2267] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:49.957038 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b086cd04a8e249a09d729be46eda67b1" } modify_external_catalogs: true
I20250109 20:52:49.957247 1158 tablet_bootstrap.cc:492] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:49.957218 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b086cd04a8e249a09d729be46eda67b1" } modify_external_catalogs: true
I20250109 20:52:49.957377 1158 ts_tablet_manager.cc:1397] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20250109 20:52:49.958087 1158 raft_consensus.cc:357] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.958282 1158 raft_consensus.cc:383] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.958364 1158 raft_consensus.cc:738] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.960805 1176 tablet_bootstrap.cc:492] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:49.964957 1158 consensus_queue.cc:260] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.965585 1158 ts_tablet_manager.cc:1428] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250109 20:52:49.970182 1176 ts_tablet_manager.cc:1397] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.241s user 0.025s sys 0.017s
I20250109 20:52:49.964659 1160 ts_tablet_manager.cc:1918] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:49.972430 1160 log.cc:1198] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/54d11b9148dc42bc91348e837af73972
I20250109 20:52:49.972851 1160 ts_tablet_manager.cc:1939] T 54d11b9148dc42bc91348e837af73972 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:49.973896 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 54d11b9148dc42bc91348e837af73972 (table default.Table_1e48be0e18b145708a9b6fa28ae21112 [id=194d1be4f3d946ec8c493ffcc5a23899]) successfully deleted
W20250109 20:52:49.974680 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cba0ffb1b394431ba30918911cdee7b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cba0ffb1b394431ba30918911cdee7b7 already in progress: creating tablet
I20250109 20:52:49.978818 748 tablet_service.cc:1467] Processing CreateTablet for tablet 9c88f42c15d94099903178218bfac1e5 (DEFAULT_TABLE table=default.Table_90ff6b4b3aed4dcf8093176305e600e5 [id=963ec7b3b46e48b18b758570e4b4b681]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.979068 1176 raft_consensus.cc:357] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.979315 1176 raft_consensus.cc:383] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:49.979416 1176 raft_consensus.cc:738] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:49.979663 1176 consensus_queue.cc:260] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:49.965261 1166 ts_tablet_manager.cc:1905] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:49.978067 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 9c88f42c15d94099903178218bfac1e5 (DEFAULT_TABLE table=default.Table_90ff6b4b3aed4dcf8093176305e600e5 [id=963ec7b3b46e48b18b758570e4b4b681]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:49.980247 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c88f42c15d94099903178218bfac1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:49.937249 1165 ts_tablet_manager.cc:1428] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:49.994215 1165 tablet_bootstrap.cc:492] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:49.996126 1165 tablet_bootstrap.cc:654] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:49.996086 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 120258us
I20250109 20:52:50.001555 1451 raft_consensus.cc:491] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.001700 1451 raft_consensus.cc:513] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.002348 1451 leader_election.cc:290] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:50.010064 1176 ts_tablet_manager.cc:1428] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250109 20:52:50.011838 1176 tablet_bootstrap.cc:492] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.023216 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 50039da210df47ab8476e28387c8a802 (DEFAULT_TABLE table=default.Table_5e6c78d0dceb4466bb81f225c45bee0d [id=b296e8173f88419a8a14ac395611b4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.023663 1166 ts_tablet_manager.cc:1918] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.024232 1166 log.cc:1198] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ffd6ce78438841daa663b6344f062019
I20250109 20:52:50.024096 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50039da210df47ab8476e28387c8a802. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.028048 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0463dd5de987418d8ea1b102b09137bf" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:50.028290 775 raft_consensus.cc:2463] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:50.030190 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db4e85cd4094cf0b5b699402346b980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.033404 1157 tablet_replica.cc:331] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:50.033574 1157 raft_consensus.cc:2238] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.033762 1157 raft_consensus.cc:2267] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.034184 1158 tablet_bootstrap.cc:492] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:50.034368 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50039da210df47ab8476e28387c8a802. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.036965 1158 tablet_bootstrap.cc:654] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.037539 1157 ts_tablet_manager.cc:1905] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.041778 1176 tablet_bootstrap.cc:654] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.042346 990 leader_election.cc:304] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:50.043105 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c88f42c15d94099903178218bfac1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.043097 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 35a39fac67894a208dd7ec3153b05e4a from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.043264 1618 raft_consensus.cc:2798] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:50.043412 1618 raft_consensus.cc:491] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:50.043550 1618 raft_consensus.cc:3054] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:50.043565 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5505cf6536bf496e88f11dc1586941d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:50.044165 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4814361a1e854451a0ee7469cc6d447b
I20250109 20:52:50.044898 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5b2fcddba2ac4d82b2d25bfc64a5967e
I20250109 20:52:50.045253 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814361a1e854451a0ee7469cc6d447b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.045779 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6a334724611e4303ba9d987033d872ba (table default.Table_c3c448aaf88a40de8c2d09f88e78442f [id=b1346293729744dda93f981b02aa3124]) successfully deleted
W20250109 20:52:50.046505 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f03d4e679a1c4793bd83cd979acced88 consensus state has no leader
I20250109 20:52:50.046721 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e with delete_type TABLET_DATA_DELETED (replaced by tablet 37ee437300414eccba9926e824f72f27) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.046950 1618 raft_consensus.cc:513] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.048070 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0463dd5de987418d8ea1b102b09137bf" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:50.048136 450 master_service.cc:946] ReplaceTablet: received request to replace tablet b405d16cc62444f2ad94fb69bb9fb6ee from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.048283 918 raft_consensus.cc:3054] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:50.048431 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.048871 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.048888 739 tablet_service.cc:1514] Processing DeleteTablet for tablet c40e5019321c497b814a281e0ab1a7f7 with delete_type TABLET_DATA_DELETED (replaced by tablet f4eb5d2e7e68498bbfa47285babedaa6) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.050570 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0463dd5de987418d8ea1b102b09137bf" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:50.050733 775 raft_consensus.cc:3054] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:50.050819 918 raft_consensus.cc:2463] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:50.051224 989 leader_election.cc:304] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:50.051542 1451 raft_consensus.cc:2798] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:49.996551 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9c88f42c15d94099903178218bfac1e5 (DEFAULT_TABLE table=default.Table_90ff6b4b3aed4dcf8093176305e600e5 [id=963ec7b3b46e48b18b758570e4b4b681]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.008349 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0463dd5de987418d8ea1b102b09137bf" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:50.051821 919 raft_consensus.cc:2371] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1. Re-sending same reply.
I20250109 20:52:50.026233 885 tablet_service.cc:1514] Processing DeleteTablet for tablet cba0ffb1b394431ba30918911cdee7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:50.052570 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5505cf6536bf496e88f11dc1586941d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:50.053527 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4814361a1e854451a0ee7469cc6d447b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4814361a1e854451a0ee7469cc6d447b already in progress: creating tablet
I20250109 20:52:50.054272 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e with delete_type TABLET_DATA_DELETED (replaced by tablet 37ee437300414eccba9926e824f72f27) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.054369 455 catalog_manager.cc:6596] ReplaceTablet: tablet 5b2fcddba2ac4d82b2d25bfc64a5967e deleted and replaced by tablet 37ee437300414eccba9926e824f72f27
I20250109 20:52:50.055588 1160 tablet_replica.cc:331] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:50.055804 1160 raft_consensus.cc:2238] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.056048 1160 raft_consensus.cc:2267] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.056625 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c88f42c15d94099903178218bfac1e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.058383 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e with delete_type TABLET_DATA_DELETED (replaced by tablet 37ee437300414eccba9926e824f72f27) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.058701 1166 ts_tablet_manager.cc:1939] T ffd6ce78438841daa663b6344f062019 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.057113 1160 ts_tablet_manager.cc:1905] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.059684 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ffd6ce78438841daa663b6344f062019 (table default.Table_d8369c071899441a9b05f8910c9743c2 [id=b939987c4a0f426b846403a8579622c7]) successfully deleted
I20250109 20:52:50.059792 1166 tablet_replica.cc:331] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:50.059968 1166 raft_consensus.cc:2238] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.060122 1166 raft_consensus.cc:2267] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:50.060158 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8890a6a93e3d48dc96955f1b51e78603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8890a6a93e3d48dc96955f1b51e78603 already in progress: creating tablet
I20250109 20:52:50.060982 1166 ts_tablet_manager.cc:1905] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.061296 1618 leader_election.cc:290] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:50.062441 1176 tablet_bootstrap.cc:492] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.062640 1176 ts_tablet_manager.cc:1397] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20250109 20:52:50.063668 1176 raft_consensus.cc:357] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.063997 1176 raft_consensus.cc:383] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.064087 1176 raft_consensus.cc:738] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.065918 1451 raft_consensus.cc:695] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:50.066149 775 raft_consensus.cc:2463] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:50.066239 1451 consensus_queue.cc:237] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.067560 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 779153d4174b4887b5dc0bb355cb4a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.067796 1176 consensus_queue.cc:260] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.068333 1176 ts_tablet_manager.cc:1428] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250109 20:52:50.068653 1176 tablet_bootstrap.cc:492] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.070274 1176 tablet_bootstrap.cc:654] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.071887 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814361a1e854451a0ee7469cc6d447b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.072960 1157 ts_tablet_manager.cc:1918] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.073091 1157 log.cc:1198] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/cba0ffb1b394431ba30918911cdee7b7
I20250109 20:52:50.074436 1157 ts_tablet_manager.cc:1939] T cba0ffb1b394431ba30918911cdee7b7 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:50.076164 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet cba0ffb1b394431ba30918911cdee7b7 (table default.Table_609a0ae37cfd4b6d8939b4829beb84d6 [id=f06f85dff8684b1cac234aa2e35b6598]) successfully deleted
W20250109 20:52:50.080531 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4870a667cd3e410590923c9e2497af54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4870a667cd3e410590923c9e2497af54 already in progress: creating tablet
W20250109 20:52:50.084573 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 527776a5d742446eb8fe61c3c855021f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 527776a5d742446eb8fe61c3c855021f already in progress: creating tablet
I20250109 20:52:50.084813 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814361a1e854451a0ee7469cc6d447b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:50.085490 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet fbe8a9e5b5fe406b988126cd1859eabc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbe8a9e5b5fe406b988126cd1859eabc already in progress: creating tablet
W20250109 20:52:50.086155 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e090f72eb1e046319b825ee5fe126332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e090f72eb1e046319b825ee5fe126332 already in progress: creating tablet
W20250109 20:52:50.086824 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ea477dc845614ea78e71e45f862b57ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea477dc845614ea78e71e45f862b57ff already in progress: creating tablet
I20250109 20:52:50.087492 1166 ts_tablet_manager.cc:1918] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.087620 1166 log.cc:1198] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/d08a3af516364157a07af7368a5f95f8
W20250109 20:52:50.087816 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b2fcddba2ac4d82b2d25bfc64a5967e already in progress: creating tablet
I20250109 20:52:50.087980 1166 ts_tablet_manager.cc:1939] T d08a3af516364157a07af7368a5f95f8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.088498 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet d08a3af516364157a07af7368a5f95f8 (table default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7 [id=ada5a5d915bf40e884cbd0e86a8e5c31]) successfully deleted
I20250109 20:52:50.090343 1158 tablet_bootstrap.cc:492] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:50.090971 1157 tablet_replica.cc:331] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:50.091162 1157 raft_consensus.cc:2238] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.091307 1157 raft_consensus.cc:2267] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.092825 1158 ts_tablet_manager.cc:1397] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.000s
W20250109 20:52:50.093231 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 527776a5d742446eb8fe61c3c855021f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 527776a5d742446eb8fe61c3c855021f already in progress: creating tablet
I20250109 20:52:50.093972 1158 raft_consensus.cc:357] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.094259 1158 raft_consensus.cc:383] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250109 20:52:50.094292 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4870a667cd3e410590923c9e2497af54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4870a667cd3e410590923c9e2497af54 already in progress: creating tablet
I20250109 20:52:50.094442 1158 raft_consensus.cc:738] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.094730 1158 consensus_queue.cc:260] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.094977 1165 tablet_bootstrap.cc:492] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:50.095140 1165 ts_tablet_manager.cc:1397] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.001s
W20250109 20:52:50.095435 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e090f72eb1e046319b825ee5fe126332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e090f72eb1e046319b825ee5fe126332 already in progress: creating tablet
I20250109 20:52:50.095764 1165 raft_consensus.cc:357] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:50.095914 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet fbe8a9e5b5fe406b988126cd1859eabc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbe8a9e5b5fe406b988126cd1859eabc already in progress: creating tablet
I20250109 20:52:50.095934 1165 raft_consensus.cc:383] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.096004 1165 raft_consensus.cc:738] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:52:50.096371 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ea477dc845614ea78e71e45f862b57ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea477dc845614ea78e71e45f862b57ff already in progress: creating tablet
I20250109 20:52:50.096643 1166 tablet_replica.cc:331] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:50.096833 1166 raft_consensus.cc:2238] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.096990 1166 raft_consensus.cc:2267] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.097455 1158 ts_tablet_manager.cc:1428] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:50.098572 1158 tablet_bootstrap.cc:492] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:50.101076 1176 tablet_bootstrap.cc:492] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.101210 1176 ts_tablet_manager.cc:1397] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250109 20:52:50.102469 1176 raft_consensus.cc:357] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.102658 1176 raft_consensus.cc:383] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.102743 1157 ts_tablet_manager.cc:1905] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.102763 1176 raft_consensus.cc:738] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.102990 1176 consensus_queue.cc:260] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.104321 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0f502d666aa043e2a0b67a4e21bcf03e
I20250109 20:52:50.104249 1165 consensus_queue.cc:260] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.104504 1176 ts_tablet_manager.cc:1428] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:50.105083 1166 ts_tablet_manager.cc:1905] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.105214 758 tablet_service.cc:1514] Processing DeleteTablet for tablet c643c1ca41e048fdab95a067f899342d with delete_type TABLET_DATA_DELETED (replaced by tablet ea5d462230734767bb010f520c86c5ab) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.105427 1165 ts_tablet_manager.cc:1428] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250109 20:52:50.105679 1165 tablet_bootstrap.cc:492] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:50.107299 1165 tablet_bootstrap.cc:654] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.108296 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.108537 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.110146 1176 tablet_bootstrap.cc:492] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.111611 1158 tablet_bootstrap.cc:654] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.113161 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d with delete_type TABLET_DATA_DELETED (replaced by tablet 3d8d67b624984a0bb327d045c55476da) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.116354 1157 ts_tablet_manager.cc:1918] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.116482 1157 log.cc:1198] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/8590e03e5d8e4f38aca6d4190a55771e
I20250109 20:52:50.118597 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d" } modify_external_catalogs: true
I20250109 20:52:50.118808 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a5c5c4a2ddf348ac8a41f935ac5400d5" } new_table_name: "default.Table_a9eb8655db18431ab35d434cfe59445c" modify_external_catalogs: true
I20250109 20:52:50.120858 1176 tablet_bootstrap.cc:654] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.121977 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6223d4bbeb7a4deebddb2936e92224c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.122422 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6223d4bbeb7a4deebddb2936e92224c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.123597 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3dbeca6078054ff0877acae499fb525c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:50.123951 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b590793190454bb09e3e7dd36515f65a" } modify_external_catalogs: true
W20250109 20:52:50.124120 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dbeca6078054ff0877acae499fb525c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.124095 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b590793190454bb09e3e7dd36515f65a" } modify_external_catalogs: true
I20250109 20:52:50.125240 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7dcbaf73ff544c3e8e804e28f8e20a6c" } new_table_name: "default.Table_6015c9f96787455ba6e143685449c213" modify_external_catalogs: true
I20250109 20:52:50.126941 1166 ts_tablet_manager.cc:1918] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.127066 1166 log.cc:1198] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f7c277dcc04d4acdbc116e41ad41d7cf
I20250109 20:52:50.127843 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.128007 1166 ts_tablet_manager.cc:1939] T f7c277dcc04d4acdbc116e41ad41d7cf P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.118827 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d" } modify_external_catalogs: true
I20250109 20:52:50.143123 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 35a39fac67894a208dd7ec3153b05e4a
I20250109 20:52:50.143720 448 catalog_manager.cc:6596] ReplaceTablet: tablet 35a39fac67894a208dd7ec3153b05e4a deleted and replaced by tablet 9629fe30752f49478afad9b93dc235eb
I20250109 20:52:50.143945 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a39fac67894a208dd7ec3153b05e4a with delete_type TABLET_DATA_DELETED (replaced by tablet 9629fe30752f49478afad9b93dc235eb) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.144163 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dcb1259a70b64057b0090a78234d47a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.144829 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcb1259a70b64057b0090a78234d47a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.146076 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b405d16cc62444f2ad94fb69bb9fb6ee
I20250109 20:52:50.146078 1160 ts_tablet_manager.cc:1918] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.146466 450 catalog_manager.cc:6596] ReplaceTablet: tablet b405d16cc62444f2ad94fb69bb9fb6ee deleted and replaced by tablet d9f18f244a344272bba2bc085686b61d
I20250109 20:52:50.146508 1160 log.cc:1198] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/cba0ffb1b394431ba30918911cdee7b7
I20250109 20:52:50.146745 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ce8c7068ce5540f5b0649980256146e3" } new_table_name: "default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483" modify_external_catalogs: true
I20250109 20:52:50.146878 1160 ts_tablet_manager.cc:1939] T cba0ffb1b394431ba30918911cdee7b7 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:50.148049 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49bb1f7179c6420c83e530237052e180" } modify_external_catalogs: true
I20250109 20:52:50.148191 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49bb1f7179c6420c83e530237052e180" } modify_external_catalogs: true
I20250109 20:52:50.150274 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee with delete_type TABLET_DATA_DELETED (replaced by tablet d9f18f244a344272bba2bc085686b61d) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.153251 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee with delete_type TABLET_DATA_DELETED (replaced by tablet d9f18f244a344272bba2bc085686b61d) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.153786 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet cba0ffb1b394431ba30918911cdee7b7 (table default.Table_609a0ae37cfd4b6d8939b4829beb84d6 [id=f06f85dff8684b1cac234aa2e35b6598]) successfully deleted
I20250109 20:52:50.154129 885 tablet_service.cc:1514] Processing DeleteTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee with delete_type TABLET_DATA_DELETED (replaced by tablet d9f18f244a344272bba2bc085686b61d) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.154198 1176 tablet_bootstrap.cc:492] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.154359 1176 ts_tablet_manager.cc:1397] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.003s
I20250109 20:52:50.155157 1176 raft_consensus.cc:357] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.155385 1176 raft_consensus.cc:383] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.155467 1176 raft_consensus.cc:738] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.155732 1176 consensus_queue.cc:260] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.157234 1176 ts_tablet_manager.cc:1428] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:50.157521 1176 tablet_bootstrap.cc:492] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.160069 1176 tablet_bootstrap.cc:654] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:50.161849 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f3c276fab6a4d068d64ea5d23c9a595 consensus state has no leader
W20250109 20:52:50.162175 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a2efcc2a6ec4b109845d8087491ca9f consensus state has no leader
I20250109 20:52:50.162240 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c344526523b540b7ae0c1d1853a60c23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:50.162456 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_354ff4ca71d54c2293728987a8eb1f88" } new_table_name: "default.Table_4c9509d9c9b04d1db58e1d8b871579d2" modify_external_catalogs: true
W20250109 20:52:50.162730 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c344526523b540b7ae0c1d1853a60c23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.163635 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ea16fcc30f2c4b2eac53f20fcebd043f
I20250109 20:52:50.165314 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.165771 885 tablet_service.cc:1514] Processing DeleteTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.168121 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_472ddec2dea04b1d83cb147944a488c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.168602 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_472ddec2dea04b1d83cb147944a488c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.169334 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.169631 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f7c277dcc04d4acdbc116e41ad41d7cf (table default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]) successfully deleted
W20250109 20:52:50.169854 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b2fcddba2ac4d82b2d25bfc64a5967e already in progress: creating tablet
W20250109 20:52:50.170063 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4814361a1e854451a0ee7469cc6d447b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4814361a1e854451a0ee7469cc6d447b already in progress: creating tablet
W20250109 20:52:50.170238 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0f502d666aa043e2a0b67a4e21bcf03e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f502d666aa043e2a0b67a4e21bcf03e already in progress: creating tablet
W20250109 20:52:50.170434 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b405d16cc62444f2ad94fb69bb9fb6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b405d16cc62444f2ad94fb69bb9fb6ee already in progress: creating tablet
W20250109 20:52:50.171061 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ea16fcc30f2c4b2eac53f20fcebd043f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea16fcc30f2c4b2eac53f20fcebd043f already in progress: creating tablet
I20250109 20:52:50.172086 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9d7587432d7345e4a225fc7d5c968c27
I20250109 20:52:50.173032 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.173753 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.173840 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.175810 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a39fac67894a208dd7ec3153b05e4a with delete_type TABLET_DATA_DELETED (replaced by tablet 9629fe30752f49478afad9b93dc235eb) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.176887 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a39fac67894a208dd7ec3153b05e4a with delete_type TABLET_DATA_DELETED (replaced by tablet 9629fe30752f49478afad9b93dc235eb) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.177512 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 46d4cad7594346dfafdee8e4c49d74e6 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.177716 1176 tablet_bootstrap.cc:492] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.177834 1176 ts_tablet_manager.cc:1397] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20250109 20:52:50.177805 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e59899b8fce64608ac583be8037520ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.178287 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e59899b8fce64608ac583be8037520ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.178471 1176 raft_consensus.cc:357] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.178668 1176 raft_consensus.cc:383] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.178742 1176 raft_consensus.cc:738] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.179329 1176 consensus_queue.cc:260] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:50.180469 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b2fcddba2ac4d82b2d25bfc64a5967e already in progress: creating tablet
I20250109 20:52:50.180691 1176 ts_tablet_manager.cc:1428] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:50.180966 1176 tablet_bootstrap.cc:492] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:50.182996 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 881ea20c17904d669c8a1fbd766ba559 consensus state has no leader
W20250109 20:52:50.185727 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9d7587432d7345e4a225fc7d5c968c27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d7587432d7345e4a225fc7d5c968c27 already in progress: creating tablet
W20250109 20:52:50.186756 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 35a39fac67894a208dd7ec3153b05e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a39fac67894a208dd7ec3153b05e4a already in progress: creating tablet
W20250109 20:52:50.186820 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0f502d666aa043e2a0b67a4e21bcf03e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f502d666aa043e2a0b67a4e21bcf03e already in progress: creating tablet
I20250109 20:52:50.187315 1160 tablet_replica.cc:331] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:50.187525 1160 raft_consensus.cc:2238] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.187674 1160 raft_consensus.cc:2267] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.191805 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:50.193272 1160 ts_tablet_manager.cc:1905] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.193447 1176 tablet_bootstrap.cc:654] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.194725 1157 ts_tablet_manager.cc:1939] T 8590e03e5d8e4f38aca6d4190a55771e P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:50.196802 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_546faac34e4d4250b0845e4886d01377"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.197304 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_546faac34e4d4250b0845e4886d01377 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.198510 1586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8328960890b142248073cec450e957d8. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:50.198733 1586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: This master's current role is: FOLLOWER
W20250109 20:52:50.199537 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4b31bd4072941d5aaf8c43dada3ccc7 consensus state has no leader
I20250109 20:52:50.199842 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 8590e03e5d8e4f38aca6d4190a55771e (table default.Table_cabbd075716c4d18aba39e549c7f0043 [id=aa04f9d0face4e1b84edffbeea2b524b]) successfully deleted
I20250109 20:52:50.200655 1632 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:52:50.200974 1632 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:50.218298 1176 tablet_bootstrap.cc:492] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.219374 1176 ts_tablet_manager.cc:1397] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.003s
I20250109 20:52:50.220052 1176 raft_consensus.cc:357] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.220271 1176 raft_consensus.cc:383] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.220332 1176 raft_consensus.cc:738] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.220541 1176 consensus_queue.cc:260] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.222335 1158 tablet_bootstrap.cc:492] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:50.222525 1158 ts_tablet_manager.cc:1397] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.124s user 0.002s sys 0.005s
I20250109 20:52:50.222599 1471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53806 (request call id 0) took 1125 ms. Trace:
I20250109 20:52:50.223570 1158 raft_consensus.cc:357] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.223874 1158 raft_consensus.cc:383] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.223991 1158 raft_consensus.cc:738] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.224273 1632 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
I20250109 20:52:50.226734 1176 ts_tablet_manager.cc:1428] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250109 20:52:50.228345 1637 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250109 20:52:50.232039 1165 tablet_bootstrap.cc:492] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:50.232241 1165 ts_tablet_manager.cc:1397] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.127s user 0.004s sys 0.001s
I20250109 20:52:50.233157 1176 tablet_bootstrap.cc:492] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.233191 1165 raft_consensus.cc:357] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.233398 1165 raft_consensus.cc:383] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.233474 1165 raft_consensus.cc:738] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.233682 1165 consensus_queue.cc:260] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.234220 1165 ts_tablet_manager.cc:1428] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:50.235934 1157 tablet_replica.cc:331] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:50.236187 1157 raft_consensus.cc:2238] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.236392 1157 raft_consensus.cc:2267] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.237833 1165 tablet_bootstrap.cc:492] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:50.237856 1176 tablet_bootstrap.cc:654] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.241611 1165 tablet_bootstrap.cc:654] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.244797 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_da940b25eb974021bef8e7d3aa57dfc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.245362 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da940b25eb974021bef8e7d3aa57dfc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.245370 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8590e03e5d8e4f38aca6d4190a55771e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.247335 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_68eba4e7c30d4954a84fbdf550b8cc07" } modify_external_catalogs: true
I20250109 20:52:50.247514 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_68eba4e7c30d4954a84fbdf550b8cc07" } modify_external_catalogs: true
I20250109 20:52:50.248108 1158 consensus_queue.cc:260] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.248178 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_897456425a2b4e48949409b18f6d770e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.248667 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_897456425a2b4e48949409b18f6d770e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.251382 1160 ts_tablet_manager.cc:1918] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.251502 1160 log.cc:1198] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/607a78e281594e9a92c6aa63ee7bdf8d
I20250109 20:52:50.251830 1160 ts_tablet_manager.cc:1939] T 607a78e281594e9a92c6aa63ee7bdf8d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:50.252566 1444 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "44d473955919435cb679edd5f8afe2c2" instance_seqno: 1736455961526528) as {username='slave'} at 127.0.104.2:33419; Asking this server to re-register.
W20250109 20:52:50.253248 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bc32ce3dff0f4d18bb17c6784b0bab41 consensus state has no leader
I20250109 20:52:50.253410 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 607a78e281594e9a92c6aa63ee7bdf8d (table default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]) successfully deleted
W20250109 20:52:50.253643 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b405d16cc62444f2ad94fb69bb9fb6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b405d16cc62444f2ad94fb69bb9fb6ee already in progress: creating tablet
W20250109 20:52:50.253867 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ea16fcc30f2c4b2eac53f20fcebd043f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea16fcc30f2c4b2eac53f20fcebd043f already in progress: creating tablet
W20250109 20:52:50.254081 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9d7587432d7345e4a225fc7d5c968c27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d7587432d7345e4a225fc7d5c968c27 already in progress: creating tablet
I20250109 20:52:50.254137 1158 ts_tablet_manager.cc:1428] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250109 20:52:50.254658 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bea81cd569304584b538f85a10ebc45b" } new_table_name: "default.Table_fef953cfacde44d38f0e2dafb4114322" modify_external_catalogs: true
I20250109 20:52:50.259008 1157 ts_tablet_manager.cc:1905] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.259052 964 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:50.261320 1443 ts_manager.cc:194] Registered new tserver with Master: 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:50.266784 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2769e5906c604bd5b14df79db765ec61 consensus state has no leader
I20250109 20:52:50.267059 1176 tablet_bootstrap.cc:492] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.267200 1176 ts_tablet_manager.cc:1397] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.003s
I20250109 20:52:50.267859 1176 raft_consensus.cc:357] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.268091 1176 raft_consensus.cc:383] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.268167 1176 raft_consensus.cc:738] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.268373 1176 consensus_queue.cc:260] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.269750 1176 ts_tablet_manager.cc:1428] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:50.269999 1176 tablet_bootstrap.cc:492] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.270449 1451 raft_consensus.cc:491] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.270557 1451 raft_consensus.cc:513] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.271037 1451 leader_election.cc:290] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:50.271536 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d981c7f13c3409f994bfdf0275e0116" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:50.272008 989 leader_election.cc:343] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.272874 1176 tablet_bootstrap.cc:654] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.274473 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 530 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:52:50.275892 453 master_service.cc:946] ReplaceTablet: received request to replace tablet b405d16cc62444f2ad94fb69bb9fb6ee from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.275879 1444 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" instance_seqno: 1736455961980804) as {username='slave'} at 127.0.104.3:49917; Asking this server to re-register.
I20250109 20:52:50.276630 1101 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:50.277297 1157 ts_tablet_manager.cc:1918] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.279018 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 530 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:52:50.280282 454 catalog_manager.cc:5526] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
I20250109 20:52:50.280681 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:50.283114 1444 ts_manager.cc:194] Registered new tserver with Master: 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:50.283665 1401 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
I20250109 20:52:50.284776 1157 log.cc:1198] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/c40e5019321c497b814a281e0ab1a7f7
I20250109 20:52:50.285226 1157 ts_tablet_manager.cc:1939] T c40e5019321c497b814a281e0ab1a7f7 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:50.290863 1632 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:52:50.292484 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d981c7f13c3409f994bfdf0275e0116" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:50.293098 990 leader_election.cc:343] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.293267 990 leader_election.cc:304] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:50.293890 1451 raft_consensus.cc:2743] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:50.294631 1158 tablet_bootstrap.cc:492] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:50.294914 1176 tablet_bootstrap.cc:492] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.295117 1176 ts_tablet_manager.cc:1397] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20250109 20:52:50.295792 1176 raft_consensus.cc:357] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.295984 1176 raft_consensus.cc:383] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.296037 1176 raft_consensus.cc:738] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.296612 1176 consensus_queue.cc:260] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.297888 1176 ts_tablet_manager.cc:1428] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:50.298156 1176 tablet_bootstrap.cc:492] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.299866 1176 tablet_bootstrap.cc:654] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.300025 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 28cd9cd48910497ba3e4b159583cb7a6
I20250109 20:52:50.300627 1443 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "09833ea46b7141d58c3899dace69e49f" instance_seqno: 1736455961029491) as {username='slave'} at 127.0.104.1:54197; Asking this server to re-register.
I20250109 20:52:50.301350 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.295140 1157 tablet_replica.cc:331] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:50.303092 1157 raft_consensus.cc:2238] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.303248 1157 raft_consensus.cc:2267] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.304397 1157 ts_tablet_manager.cc:1905] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.307974 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.308871 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet c40e5019321c497b814a281e0ab1a7f7 (table default.Table_579b6c1d61dd4e80af5ea30b4e06dc35 [id=579828ccd3744a63b1212b6de38038ce]) successfully deleted
W20250109 20:52:50.309394 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 28cd9cd48910497ba3e4b159583cb7a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28cd9cd48910497ba3e4b159583cb7a6 already in progress: creating tablet
I20250109 20:52:50.309473 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 829bec4f66c24180afd5271057a7d0e8
I20250109 20:52:50.310593 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 829bec4f66c24180afd5271057a7d0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.311246 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 829bec4f66c24180afd5271057a7d0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.311846 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5aceb6f20bb841acb677692231b8f670" } modify_external_catalogs: true
I20250109 20:52:50.312048 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5aceb6f20bb841acb677692231b8f670" } modify_external_catalogs: true
I20250109 20:52:50.313753 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 829bec4f66c24180afd5271057a7d0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.313966 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:50.314438 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 829bec4f66c24180afd5271057a7d0e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829bec4f66c24180afd5271057a7d0e8 already in progress: creating tablet
I20250109 20:52:50.314646 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 607a78e281594e9a92c6aa63ee7bdf8d with delete_type TABLET_DATA_DELETED (replaced by tablet 3d8d67b624984a0bb327d045c55476da) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.315213 823 heartbeater.cc:463] Registering TS with master...
I20250109 20:52:50.318702 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 72a6cf525cb44c428f7a39a25c4e62dc
I20250109 20:52:50.319844 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a6cf525cb44c428f7a39a25c4e62dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.320174 1176 tablet_bootstrap.cc:492] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.320158 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a6cf525cb44c428f7a39a25c4e62dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.320288 1176 ts_tablet_manager.cc:1397] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20250109 20:52:50.320708 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a6cf525cb44c428f7a39a25c4e62dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.321003 1176 raft_consensus.cc:357] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.321179 1176 raft_consensus.cc:383] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.321252 1176 raft_consensus.cc:738] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.321272 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
W20250109 20:52:50.321506 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 72a6cf525cb44c428f7a39a25c4e62dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a6cf525cb44c428f7a39a25c4e62dc already in progress: creating tablet
I20250109 20:52:50.321447 1176 consensus_queue.cc:260] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.322427 1157 ts_tablet_manager.cc:1918] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.322558 1157 log.cc:1198] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/779153d4174b4887b5dc0bb355cb4a9d
I20250109 20:52:50.322844 1176 ts_tablet_manager.cc:1428] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:50.322876 1157 ts_tablet_manager.cc:1939] T 779153d4174b4887b5dc0bb355cb4a9d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:50.323119 1176 tablet_bootstrap.cc:492] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.324322 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 with delete_type TABLET_DATA_DELETED (replaced by tablet e64207928ef141118e7fc8fe7bb2379b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.324549 1166 tablet_replica.cc:331] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:50.324766 1166 raft_consensus.cc:2238] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.324918 1166 raft_consensus.cc:2267] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.326013 1166 ts_tablet_manager.cc:1905] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.326182 1176 tablet_bootstrap.cc:654] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.328940 1157 tablet_replica.cc:331] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:50.329157 1157 raft_consensus.cc:2238] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.329344 1157 raft_consensus.cc:2267] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.330003 890 tablet_service.cc:1467] Processing CreateTablet for tablet 12e97ff530df46ae80eb1b5615a55a3f (DEFAULT_TABLE table=default.Table_60c85531239c48349f1931cbcf6b2977 [id=73a9141d6eb942018d73d2c01d601801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.330302 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e97ff530df46ae80eb1b5615a55a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.331130 1158 tablet_bootstrap.cc:654] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.333017 1160 tablet_replica.cc:331] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:50.333216 1160 raft_consensus.cc:2238] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.333393 1160 raft_consensus.cc:2267] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.333603 895 tablet_service.cc:1467] Processing CreateTablet for tablet a1ee36f3be324b14868976298ca26118 (DEFAULT_TABLE table=default.Table_f8eaab37b5024ccd99aa643e72cdda0a [id=ac6293f1f6fe4760ae7f07e3c925081f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.333979 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ee36f3be324b14868976298ca26118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.334133 1160 ts_tablet_manager.cc:1905] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.334451 888 tablet_service.cc:1467] Processing CreateTablet for tablet 75cdf5b737504d4e8f598cd2b265192e (DEFAULT_TABLE table=default.Table_98b8f1400db644be881e04403ed16d10 [id=a4138fd6ff5246d582fc0d75b890c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.334769 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75cdf5b737504d4e8f598cd2b265192e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.333410 898 tablet_service.cc:1467] Processing CreateTablet for tablet 374df8c6d1424697a3b3922f777e9870 (DEFAULT_TABLE table=default.Table_9ef276a3879c406abf2b9f0a326cde2a [id=f1cdb4acec9a469cafbd3e517556a3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.337486 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374df8c6d1424697a3b3922f777e9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.333017 885 tablet_service.cc:1467] Processing CreateTablet for tablet b56d2c0706c84195acceb97ef4985c14 (DEFAULT_TABLE table=default.Table_e75baed496db483c91ad7464f1678e0d [id=3064576ec4fb44338fa6ed3200e18ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.341135 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56d2c0706c84195acceb97ef4985c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.342432 1158 tablet_bootstrap.cc:492] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:50.342604 1158 ts_tablet_manager.cc:1397] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.003s
I20250109 20:52:50.343425 1158 raft_consensus.cc:357] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.344341 1158 raft_consensus.cc:383] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.344434 1158 raft_consensus.cc:738] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.344682 1160 ts_tablet_manager.cc:1918] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.344813 1160 log.cc:1198] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/8590e03e5d8e4f38aca6d4190a55771e
I20250109 20:52:50.344833 1158 consensus_queue.cc:260] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.345139 1160 ts_tablet_manager.cc:1939] T 8590e03e5d8e4f38aca6d4190a55771e P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:50.345417 1158 ts_tablet_manager.cc:1428] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250109 20:52:50.347579 884 tablet_service.cc:1467] Processing CreateTablet for tablet 4f90c364ce844c9ea8eccb4ae23f8dce (DEFAULT_TABLE table=default.Table_fca16c47746e4cbeb149a7d20fd0a979 [id=2a02c7d57a6c4df7ac423f494891b8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.347929 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f90c364ce844c9ea8eccb4ae23f8dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.349049 880 tablet_service.cc:1467] Processing CreateTablet for tablet 31de9638669040db9674d3aa7df32103 (DEFAULT_TABLE table=default.Table_df97156287614f04b00e64a3ecfb64ce [id=7afdb751b30e46948bdae5ca45b684d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.349378 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31de9638669040db9674d3aa7df32103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.351030 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7de47568081c4d11a33a67edd6898f8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.351498 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7de47568081c4d11a33a67edd6898f8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.352277 886 tablet_service.cc:1467] Processing CreateTablet for tablet c70ddcc69b2b42fcbe133978dd3ac05b (DEFAULT_TABLE table=default.Table_e93d193814ee46d68c0d48c550964f48 [id=102d590adc1f44028550f3508da63869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.352545 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70ddcc69b2b42fcbe133978dd3ac05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.354027 1158 tablet_bootstrap.cc:492] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:50.355504 896 tablet_service.cc:1467] Processing CreateTablet for tablet 9d13168ca13c400a8a2d9608293ff8c3 (DEFAULT_TABLE table=default.Table_ccc647e63d744a0cbfd48bdbd78f5f87 [id=c6c157e5119d4f558b3e2310d2dc3f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.355782 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d13168ca13c400a8a2d9608293ff8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.356949 893 tablet_service.cc:1467] Processing CreateTablet for tablet beca5f1ed9d3434fb7de90cd74a6f09b (DEFAULT_TABLE table=default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.357223 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beca5f1ed9d3434fb7de90cd74a6f09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.357311 1158 tablet_bootstrap.cc:654] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.360317 897 tablet_service.cc:1467] Processing CreateTablet for tablet e457baa02f4e48f8a627ffdb05267fc8 (DEFAULT_TABLE table=default.Table_a1bedae7fe314ca1bb17950aa1914398 [id=c7ac68af36ab444ea36032ce72130972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.360616 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e457baa02f4e48f8a627ffdb05267fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.365672 1176 tablet_bootstrap.cc:492] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.365814 1176 ts_tablet_manager.cc:1397] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.043s user 0.024s sys 0.013s
I20250109 20:52:50.366462 1176 raft_consensus.cc:357] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.366653 1176 raft_consensus.cc:383] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.366729 1176 raft_consensus.cc:738] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.366930 1176 consensus_queue.cc:260] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.367445 1176 ts_tablet_manager.cc:1428] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:50.367753 1176 tablet_bootstrap.cc:492] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.369040 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c2a802cfb927494f92e603090370e543" } new_table_name: "default.Table_d2ee28b720884b02a77177a6c829d85a" modify_external_catalogs: true
I20250109 20:52:50.370103 1176 tablet_bootstrap.cc:654] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.370325 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8de32e389cf24db59b472ed2bbabb3c0" } new_table_name: "default.Table_17e6c9ac41454504bfb114714d0a9485" modify_external_catalogs: true
I20250109 20:52:50.371340 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4" } modify_external_catalogs: true
I20250109 20:52:50.371508 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4" } modify_external_catalogs: true
I20250109 20:52:50.372004 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_283db7be437c43d8845b9f6d4d906bc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.372406 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_283db7be437c43d8845b9f6d4d906bc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:50.377063 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 35a39fac67894a208dd7ec3153b05e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a39fac67894a208dd7ec3153b05e4a already in progress: creating tablet
I20250109 20:52:50.377249 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 8590e03e5d8e4f38aca6d4190a55771e (table default.Table_cabbd075716c4d18aba39e549c7f0043 [id=aa04f9d0face4e1b84edffbeea2b524b]) successfully deleted
W20250109 20:52:50.377420 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 28cd9cd48910497ba3e4b159583cb7a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28cd9cd48910497ba3e4b159583cb7a6 already in progress: creating tablet
W20250109 20:52:50.377599 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 829bec4f66c24180afd5271057a7d0e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829bec4f66c24180afd5271057a7d0e8 already in progress: creating tablet
I20250109 20:52:50.377691 1158 tablet_bootstrap.cc:492] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
W20250109 20:52:50.377779 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 72a6cf525cb44c428f7a39a25c4e62dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a6cf525cb44c428f7a39a25c4e62dc already in progress: creating tablet
I20250109 20:52:50.377844 1158 ts_tablet_manager.cc:1397] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250109 20:52:50.378751 1158 raft_consensus.cc:357] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.379019 1158 raft_consensus.cc:383] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.379113 1158 raft_consensus.cc:738] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.379362 1158 consensus_queue.cc:260] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.379971 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_51b311b735684e74a05c0cd8973a089e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.380384 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51b311b735684e74a05c0cd8973a089e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.382047 1158 ts_tablet_manager.cc:1428] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:50.382326 1158 tablet_bootstrap.cc:492] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:50.385174 1158 tablet_bootstrap.cc:654] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:50.385661 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a45cd5ffe7040e385d8cdfb97ed011a consensus state has no leader
W20250109 20:52:50.386058 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fb563bd6802473fafedd08bb10d15c1 consensus state has no leader
I20250109 20:52:50.399864 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6c79e0eceb7b4c6a8c751387a1a7118a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.400398 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c79e0eceb7b4c6a8c751387a1a7118a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.401965 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fca16c47746e4cbeb149a7d20fd0a979" } modify_external_catalogs: true
I20250109 20:52:50.402127 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fca16c47746e4cbeb149a7d20fd0a979" } modify_external_catalogs: true
I20250109 20:52:50.402879 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6b76c70db0cd45aa9579e976215b1470" } new_table_name: "default.Table_9f1c5bbd4e254d71b52e77fbaadb8b8b" modify_external_catalogs: true
I20250109 20:52:50.404515 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9498c3dc02794da496271a51706a9c7a" } new_table_name: "default.Table_9a67ee66b9a049a78f9a0788ec87dd26" modify_external_catalogs: true
I20250109 20:52:50.404578 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5cf48a44b13a4059b1e8974875e736e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.405040 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cf48a44b13a4059b1e8974875e736e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.405342 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6c195088c8764cc2a94609872588c4a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.405726 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c195088c8764cc2a94609872588c4a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.416200 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2bdba403f4a24ce392324eeb74d3ed0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.416613 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bdba403f4a24ce392324eeb74d3ed0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.428650 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9a6081630d9c4441916bca2f2d06ce63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.429136 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a6081630d9c4441916bca2f2d06ce63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.433102 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:50.436249 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:50.446442 1166 ts_tablet_manager.cc:1918] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.452153 1157 ts_tablet_manager.cc:1905] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.458076 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 779153d4174b4887b5dc0bb355cb4a9d (table default.Table_0f1b6adda4ab4df18b08fb9f430907e8 [id=6af19c1506c9416d979c0d938080d827]) successfully deleted
W20250109 20:52:50.458835 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4814361a1e854451a0ee7469cc6d447b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4814361a1e854451a0ee7469cc6d447b already in progress: creating tablet
I20250109 20:52:50.460657 748 tablet_service.cc:1467] Processing CreateTablet for tablet 12e97ff530df46ae80eb1b5615a55a3f (DEFAULT_TABLE table=default.Table_60c85531239c48349f1931cbcf6b2977 [id=73a9141d6eb942018d73d2c01d601801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.461058 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e97ff530df46ae80eb1b5615a55a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.461654 1443 ts_manager.cc:194] Registered new tserver with Master: 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:50.462301 757 tablet_service.cc:1467] Processing CreateTablet for tablet 374df8c6d1424697a3b3922f777e9870 (DEFAULT_TABLE table=default.Table_9ef276a3879c406abf2b9f0a326cde2a [id=f1cdb4acec9a469cafbd3e517556a3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.462600 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374df8c6d1424697a3b3922f777e9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.464999 758 tablet_service.cc:1467] Processing CreateTablet for tablet b56d2c0706c84195acceb97ef4985c14 (DEFAULT_TABLE table=default.Table_e75baed496db483c91ad7464f1678e0d [id=3064576ec4fb44338fa6ed3200e18ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.465330 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56d2c0706c84195acceb97ef4985c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.470551 757 tablet_service.cc:1467] Processing CreateTablet for tablet a1ee36f3be324b14868976298ca26118 (DEFAULT_TABLE table=default.Table_f8eaab37b5024ccd99aa643e72cdda0a [id=ac6293f1f6fe4760ae7f07e3c925081f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.471062 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ee36f3be324b14868976298ca26118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.474360 1165 tablet_bootstrap.cc:492] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:50.474510 1165 ts_tablet_manager.cc:1397] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.237s user 0.003s sys 0.000s
I20250109 20:52:50.457204 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e8ee4b86a04199865acd5f69cda384 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.457602 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.457928 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a9910f88574eac90db9e00138c8dda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.459404 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6716e7428ce240fca7ca5b02f564a5df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.459690 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet c03524194a7b4e34a20424091e308ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.475369 1165 raft_consensus.cc:357] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.460018 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 12e97ff530df46ae80eb1b5615a55a3f (DEFAULT_TABLE table=default.Table_60c85531239c48349f1931cbcf6b2977 [id=73a9141d6eb942018d73d2c01d601801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.475562 1165 raft_consensus.cc:383] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.475638 1165 raft_consensus.cc:738] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.460901 1032 tablet_service.cc:1467] Processing CreateTablet for tablet b56d2c0706c84195acceb97ef4985c14 (DEFAULT_TABLE table=default.Table_e75baed496db483c91ad7464f1678e0d [id=3064576ec4fb44338fa6ed3200e18ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.475845 1165 consensus_queue.cc:260] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.476204 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12e97ff530df46ae80eb1b5615a55a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.476297 1165 ts_tablet_manager.cc:1428] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:50.476536 1165 tablet_bootstrap.cc:492] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:50.461745 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 374df8c6d1424697a3b3922f777e9870 (DEFAULT_TABLE table=default.Table_9ef276a3879c406abf2b9f0a326cde2a [id=f1cdb4acec9a469cafbd3e517556a3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.477898 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374df8c6d1424697a3b3922f777e9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.470382 1021 tablet_service.cc:1467] Processing CreateTablet for tablet a1ee36f3be324b14868976298ca26118 (DEFAULT_TABLE table=default.Table_f8eaab37b5024ccd99aa643e72cdda0a [id=ac6293f1f6fe4760ae7f07e3c925081f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.480614 1165 tablet_bootstrap.cc:654] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.482769 744 tablet_service.cc:1467] Processing CreateTablet for tablet 31de9638669040db9674d3aa7df32103 (DEFAULT_TABLE table=default.Table_df97156287614f04b00e64a3ecfb64ce [id=7afdb751b30e46948bdae5ca45b684d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.483151 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31de9638669040db9674d3aa7df32103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.483094 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 75cdf5b737504d4e8f598cd2b265192e (DEFAULT_TABLE table=default.Table_98b8f1400db644be881e04403ed16d10 [id=a4138fd6ff5246d582fc0d75b890c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.483402 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ee36f3be324b14868976298ca26118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.487982 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 31de9638669040db9674d3aa7df32103 (DEFAULT_TABLE table=default.Table_df97156287614f04b00e64a3ecfb64ce [id=7afdb751b30e46948bdae5ca45b684d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.488618 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 4f90c364ce844c9ea8eccb4ae23f8dce (DEFAULT_TABLE table=default.Table_fca16c47746e4cbeb149a7d20fd0a979 [id=2a02c7d57a6c4df7ac423f494891b8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.488987 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56d2c0706c84195acceb97ef4985c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.489889 748 tablet_service.cc:1467] Processing CreateTablet for tablet 75cdf5b737504d4e8f598cd2b265192e (DEFAULT_TABLE table=default.Table_98b8f1400db644be881e04403ed16d10 [id=a4138fd6ff5246d582fc0d75b890c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.490219 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75cdf5b737504d4e8f598cd2b265192e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.493968 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31de9638669040db9674d3aa7df32103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.494637 739 tablet_service.cc:1467] Processing CreateTablet for tablet 9d13168ca13c400a8a2d9608293ff8c3 (DEFAULT_TABLE table=default.Table_ccc647e63d744a0cbfd48bdbd78f5f87 [id=c6c157e5119d4f558b3e2310d2dc3f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.495003 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d13168ca13c400a8a2d9608293ff8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.495296 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d11b9148dc42bc91348e837af73972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.495398 744 tablet_service.cc:1467] Processing CreateTablet for tablet c70ddcc69b2b42fcbe133978dd3ac05b (DEFAULT_TABLE table=default.Table_e93d193814ee46d68c0d48c550964f48 [id=102d590adc1f44028550f3508da63869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.495731 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70ddcc69b2b42fcbe133978dd3ac05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.495664 751 tablet_service.cc:1467] Processing CreateTablet for tablet e457baa02f4e48f8a627ffdb05267fc8 (DEFAULT_TABLE table=default.Table_a1bedae7fe314ca1bb17950aa1914398 [id=c7ac68af36ab444ea36032ce72130972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.496147 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e457baa02f4e48f8a627ffdb05267fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.496955 749 tablet_service.cc:1467] Processing CreateTablet for tablet beca5f1ed9d3434fb7de90cd74a6f09b (DEFAULT_TABLE table=default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.497251 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beca5f1ed9d3434fb7de90cd74a6f09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.509168 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75cdf5b737504d4e8f598cd2b265192e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.510429 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f90c364ce844c9ea8eccb4ae23f8dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.514442 1033 tablet_service.cc:1467] Processing CreateTablet for tablet c70ddcc69b2b42fcbe133978dd3ac05b (DEFAULT_TABLE table=default.Table_e93d193814ee46d68c0d48c550964f48 [id=102d590adc1f44028550f3508da63869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.515096 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 9d13168ca13c400a8a2d9608293ff8c3 (DEFAULT_TABLE table=default.Table_ccc647e63d744a0cbfd48bdbd78f5f87 [id=c6c157e5119d4f558b3e2310d2dc3f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.515872 1027 tablet_service.cc:1467] Processing CreateTablet for tablet beca5f1ed9d3434fb7de90cd74a6f09b (DEFAULT_TABLE table=default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.521344 1026 tablet_service.cc:1467] Processing CreateTablet for tablet e457baa02f4e48f8a627ffdb05267fc8 (DEFAULT_TABLE table=default.Table_a1bedae7fe314ca1bb17950aa1914398 [id=c7ac68af36ab444ea36032ce72130972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.528868 1590 raft_consensus.cc:491] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.529018 1590 raft_consensus.cc:513] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.529699 1590 leader_election.cc:290] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:50.530676 712 leader_election.cc:343] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Not found: Tablet not found: 607a78e281594e9a92c6aa63ee7bdf8d
I20250109 20:52:50.534801 1165 tablet_bootstrap.cc:492] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:50.539258 1165 ts_tablet_manager.cc:1397] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.007s
I20250109 20:52:50.540354 1165 raft_consensus.cc:357] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.541595 1165 raft_consensus.cc:383] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.541857 1165 raft_consensus.cc:738] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.542295 1165 consensus_queue.cc:260] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.547753 1165 ts_tablet_manager.cc:1428] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250109 20:52:50.549644 758 tablet_service.cc:1467] Processing CreateTablet for tablet 4f90c364ce844c9ea8eccb4ae23f8dce (DEFAULT_TABLE table=default.Table_fca16c47746e4cbeb149a7d20fd0a979 [id=2a02c7d57a6c4df7ac423f494891b8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.550208 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f90c364ce844c9ea8eccb4ae23f8dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.552433 1165 tablet_bootstrap.cc:492] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:50.554630 1165 tablet_bootstrap.cc:654] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.558679 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70ddcc69b2b42fcbe133978dd3ac05b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:50.558750 713 leader_election.cc:343] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 607a78e281594e9a92c6aa63ee7bdf8d
I20250109 20:52:50.558912 713 leader_election.cc:304] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:50.559439 1590 raft_consensus.cc:2743] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:50.566139 1165 tablet_bootstrap.cc:492] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:50.566712 1165 ts_tablet_manager.cc:1397] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250109 20:52:50.567828 1165 raft_consensus.cc:357] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.569021 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beca5f1ed9d3434fb7de90cd74a6f09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.569216 1165 raft_consensus.cc:383] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.570864 1165 raft_consensus.cc:738] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.571310 1165 consensus_queue.cc:260] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.572954 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e457baa02f4e48f8a627ffdb05267fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.576350 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d13168ca13c400a8a2d9608293ff8c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.576620 1165 ts_tablet_manager.cc:1428] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250109 20:52:50.581703 1165 tablet_bootstrap.cc:492] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:50.585629 1165 tablet_bootstrap.cc:654] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.586750 1166 log.cc:1198] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2b9feb9de9dc4d22bafc0f7707943113
I20250109 20:52:50.587177 1166 ts_tablet_manager.cc:1939] T 2b9feb9de9dc4d22bafc0f7707943113 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.588178 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2b9feb9de9dc4d22bafc0f7707943113 (table default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]) successfully deleted
I20250109 20:52:50.596261 1166 tablet_replica.cc:331] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:50.596509 1166 raft_consensus.cc:2238] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.596539 1191 raft_consensus.cc:491] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.596791 1166 raft_consensus.cc:2267] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.596670 1191 raft_consensus.cc:513] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.597265 1191 leader_election.cc:290] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:50.597672 1451 raft_consensus.cc:491] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.597771 1166 ts_tablet_manager.cc:1905] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.597816 1451 raft_consensus.cc:513] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.597903 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a9910f88574eac90db9e00138c8dda" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:50.597959 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a9910f88574eac90db9e00138c8dda" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:50.598119 1054 raft_consensus.cc:2463] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:50.598376 1451 leader_election.cc:290] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:50.598506 854 leader_election.cc:343] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.598677 854 leader_election.cc:304] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f
I20250109 20:52:50.599031 1191 raft_consensus.cc:2798] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:50.599128 1191 raft_consensus.cc:491] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:50.599213 1191 raft_consensus.cc:3054] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:50.599239 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f672cbd6dffc49dbb69992ae7db979ea" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:50.599414 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f672cbd6dffc49dbb69992ae7db979ea" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:50.599603 775 raft_consensus.cc:2463] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
W20250109 20:52:50.600003 989 leader_election.cc:343] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.600376 990 leader_election.cc:304] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:50.601043 1451 raft_consensus.cc:2798] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:50.601145 1451 raft_consensus.cc:491] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:50.601244 1451 raft_consensus.cc:3054] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:50.601699 1191 raft_consensus.cc:513] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.602257 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:50.603152 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a9910f88574eac90db9e00138c8dda" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
W20250109 20:52:50.603552 854 leader_election.cc:343] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.603819 1191 leader_election.cc:290] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:50.603735 1451 raft_consensus.cc:513] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.604368 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a9910f88574eac90db9e00138c8dda" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:50.604566 1054 raft_consensus.cc:3054] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:50.604904 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f672cbd6dffc49dbb69992ae7db979ea" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:50.605085 775 raft_consensus.cc:3054] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:50.605147 1165 tablet_bootstrap.cc:492] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:50.605798 1165 ts_tablet_manager.cc:1397] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250109 20:52:50.606422 1165 raft_consensus.cc:357] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.606585 1165 raft_consensus.cc:383] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.606632 1165 raft_consensus.cc:738] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.606925 1054 raft_consensus.cc:2463] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:50.607290 854 leader_election.cc:304] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f
I20250109 20:52:50.607626 1191 raft_consensus.cc:2798] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:50.607685 895 tablet_service.cc:1467] Processing CreateTablet for tablet f8eba3dcab764a98857604b63bf9dce1 (DEFAULT_TABLE table=default.Table_e59899b8fce64608ac583be8037520ca [id=211bbf02224e419caabe39e274c38b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.607571 775 raft_consensus.cc:2463] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:50.608085 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8eba3dcab764a98857604b63bf9dce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.608249 990 leader_election.cc:304] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:50.609741 1451 leader_election.cc:290] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:50.609664 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f672cbd6dffc49dbb69992ae7db979ea" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:50.610173 989 leader_election.cc:343] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.610469 1451 raft_consensus.cc:2798] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:50.610787 1451 raft_consensus.cc:695] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:50.611127 1451 consensus_queue.cc:237] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.617413 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f456c719d70549da87d54647fd2d02aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.617965 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f456c719d70549da87d54647fd2d02aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.619318 749 tablet_service.cc:1467] Processing CreateTablet for tablet f8eba3dcab764a98857604b63bf9dce1 (DEFAULT_TABLE table=default.Table_e59899b8fce64608ac583be8037520ca [id=211bbf02224e419caabe39e274c38b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.619726 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8eba3dcab764a98857604b63bf9dce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.624332 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:50.610786 1166 ts_tablet_manager.cc:1918] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.624485 1166 log.cc:1198] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f0e8ee4b86a04199865acd5f69cda384
I20250109 20:52:50.625489 1166 ts_tablet_manager.cc:1939] T f0e8ee4b86a04199865acd5f69cda384 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.626346 1166 tablet_replica.cc:331] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:50.626340 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0538f4d3608a454b9dd21a0fa42068a0 consensus state has no leader
I20250109 20:52:50.626560 1166 raft_consensus.cc:2238] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.626703 1166 raft_consensus.cc:2267] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.627457 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a334724611e4303ba9d987033d872ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.627686 1160 tablet_replica.cc:331] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:50.627861 1160 raft_consensus.cc:2238] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.628018 1160 raft_consensus.cc:2267] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.628576 1160 ts_tablet_manager.cc:1905] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.631000 1191 raft_consensus.cc:695] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Running, Role: LEADER
I20250109 20:52:50.631315 1191 consensus_queue.cc:237] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.634289 1165 consensus_queue.cc:260] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.635663 1160 ts_tablet_manager.cc:1918] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.635848 1160 log.cc:1198] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/6a334724611e4303ba9d987033d872ba
I20250109 20:52:50.635902 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f0e8ee4b86a04199865acd5f69cda384 (table default.Table_b683408175de45c0bf7d7f7e14e15447 [id=2a713bcff6ae4df0a2a9ad75a6986394]) successfully deleted
I20250109 20:52:50.636230 1160 ts_tablet_manager.cc:1939] T 6a334724611e4303ba9d987033d872ba P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:50.636788 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 6a334724611e4303ba9d987033d872ba (table default.Table_c3c448aaf88a40de8c2d09f88e78442f [id=b1346293729744dda93f981b02aa3124]) successfully deleted
I20250109 20:52:50.637682 1020 tablet_service.cc:1467] Processing CreateTablet for tablet f8eba3dcab764a98857604b63bf9dce1 (DEFAULT_TABLE table=default.Table_e59899b8fce64608ac583be8037520ca [id=211bbf02224e419caabe39e274c38b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.635685 1165 ts_tablet_manager.cc:1428] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250109 20:52:50.638223 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8eba3dcab764a98857604b63bf9dce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.638345 1166 ts_tablet_manager.cc:1905] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.642206 1165 tablet_bootstrap.cc:492] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:50.642354 1157 ts_tablet_manager.cc:1918] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.642506 1157 log.cc:1198] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/c643c1ca41e048fdab95a067f899342d
I20250109 20:52:50.642899 1157 ts_tablet_manager.cc:1939] T c643c1ca41e048fdab95a067f899342d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:50.644017 1165 tablet_bootstrap.cc:654] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.654186 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet c643c1ca41e048fdab95a067f899342d (table default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]) successfully deleted
I20250109 20:52:50.674432 886 tablet_service.cc:1467] Processing CreateTablet for tablet f2383d9770834b1aae290308b4185299 (DEFAULT_TABLE table=default.Table_c344526523b540b7ae0c1d1853a60c23 [id=51bb90fe67424c6fa1320fe03540875a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.675141 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2383d9770834b1aae290308b4185299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.675465 1451 raft_consensus.cc:491] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.675634 1451 raft_consensus.cc:513] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.676267 1451 leader_election.cc:290] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:50.676952 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6716e7428ce240fca7ca5b02f564a5df" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:50.678092 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6716e7428ce240fca7ca5b02f564a5df" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:50.678233 990 leader_election.cc:343] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:50.678658 989 leader_election.cc:343] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.678807 989 leader_election.cc:304] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:50.679314 1451 raft_consensus.cc:2743] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:50.681798 1166 ts_tablet_manager.cc:1918] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.681864 757 tablet_service.cc:1467] Processing CreateTablet for tablet f2383d9770834b1aae290308b4185299 (DEFAULT_TABLE table=default.Table_c344526523b540b7ae0c1d1853a60c23 [id=51bb90fe67424c6fa1320fe03540875a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.682266 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2383d9770834b1aae290308b4185299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.685802 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f2383d9770834b1aae290308b4185299 (DEFAULT_TABLE table=default.Table_c344526523b540b7ae0c1d1853a60c23 [id=51bb90fe67424c6fa1320fe03540875a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.686525 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2383d9770834b1aae290308b4185299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.689332 757 tablet_service.cc:1467] Processing CreateTablet for tablet 7df7405e93aa4718a62933ae56af9f42 (DEFAULT_TABLE table=default.Table_6223d4bbeb7a4deebddb2936e92224c0 [id=f88d49eba7374387915b697342d8ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.689658 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df7405e93aa4718a62933ae56af9f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.694243 1166 log.cc:1198] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/7d981c7f13c3409f994bfdf0275e0116
I20250109 20:52:50.696470 1166 ts_tablet_manager.cc:1939] T 7d981c7f13c3409f994bfdf0275e0116 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.698086 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_24b4edbc6e754ee1aeb45736c5b4aa8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:50.698392 455 catalog_manager.cc:5526] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
W20250109 20:52:50.698681 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24b4edbc6e754ee1aeb45736c5b4aa8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.698819 455 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:50.699568 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_17e6c9ac41454504bfb114714d0a9485" } new_table_name: "default.Table_ff758d5ff9a64f8db3450cff1aea31db" modify_external_catalogs: true
I20250109 20:52:50.701601 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c07faa9c0a64412588c149655620702d" } modify_external_catalogs: true
I20250109 20:52:50.701777 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c07faa9c0a64412588c149655620702d" } modify_external_catalogs: true
I20250109 20:52:50.702050 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 7d981c7f13c3409f994bfdf0275e0116 (table default.Table_2acc983c759e437cbce5b1a3a5e88cfe [id=b64c15100a8344a6bc35a896aec8270c]) successfully deleted
I20250109 20:52:50.702077 884 tablet_service.cc:1467] Processing CreateTablet for tablet 7df7405e93aa4718a62933ae56af9f42 (DEFAULT_TABLE table=default.Table_6223d4bbeb7a4deebddb2936e92224c0 [id=f88d49eba7374387915b697342d8ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.702487 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df7405e93aa4718a62933ae56af9f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.704361 1166 tablet_replica.cc:331] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:50.704602 1166 raft_consensus.cc:2238] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.705029 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 7df7405e93aa4718a62933ae56af9f42 (DEFAULT_TABLE table=default.Table_6223d4bbeb7a4deebddb2936e92224c0 [id=f88d49eba7374387915b697342d8ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.705557 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df7405e93aa4718a62933ae56af9f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.710628 1166 raft_consensus.cc:2267] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.714573 1166 ts_tablet_manager.cc:1905] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.724251 1176 tablet_bootstrap.cc:492] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.724408 1176 ts_tablet_manager.cc:1397] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.357s user 0.073s sys 0.195s
I20250109 20:52:50.724576 756 tablet_service.cc:1467] Processing CreateTablet for tablet 44af81df98ae4b05927dab7b4246de9f (DEFAULT_TABLE table=default.Table_5505cf6536bf496e88f11dc1586941d0 [id=e802914008204d5c9bbe7732f3c28fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.725061 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44af81df98ae4b05927dab7b4246de9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.725248 1176 raft_consensus.cc:357] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.725438 1176 raft_consensus.cc:383] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.725524 1176 raft_consensus.cc:738] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.725795 1176 consensus_queue.cc:260] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.729584 1176 ts_tablet_manager.cc:1428] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:50.733021 1176 tablet_bootstrap.cc:492] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.734916 1176 tablet_bootstrap.cc:654] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.736483 1451 raft_consensus.cc:491] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.736665 1451 raft_consensus.cc:513] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.737277 1451 leader_election.cc:290] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:50.737805 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00aaf81b34445abb7502700f01d4f3f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:50.737907 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00aaf81b34445abb7502700f01d4f3f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:50.738359 990 leader_election.cc:343] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:50.738492 989 leader_election.cc:343] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.738659 989 leader_election.cc:304] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:50.739053 1451 raft_consensus.cc:2743] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:50.739501 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 44af81df98ae4b05927dab7b4246de9f (DEFAULT_TABLE table=default.Table_5505cf6536bf496e88f11dc1586941d0 [id=e802914008204d5c9bbe7732f3c28fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.739904 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44af81df98ae4b05927dab7b4246de9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.749490 884 tablet_service.cc:1467] Processing CreateTablet for tablet 44af81df98ae4b05927dab7b4246de9f (DEFAULT_TABLE table=default.Table_5505cf6536bf496e88f11dc1586941d0 [id=e802914008204d5c9bbe7732f3c28fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.749928 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44af81df98ae4b05927dab7b4246de9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.750947 1166 ts_tablet_manager.cc:1918] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:50.757347 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0f502d666aa043e2a0b67a4e21bcf03e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f502d666aa043e2a0b67a4e21bcf03e already in progress: creating tablet
I20250109 20:52:50.222800 1471 rpcz_store.cc:276] 0109 20:52:49.097194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:49.097247 (+ 53us) service_pool.cc:225] Handling call
0109 20:52:49.533343 (+436096us) raft_consensus.cc:1466] Updating replica for 484 ops
0109 20:52:49.629433 (+ 96090us) raft_consensus.cc:1529] Early marking committed up to index 0
0109 20:52:49.629444 (+ 11us) raft_consensus.cc:1534] Triggering prepare for 484 ops
0109 20:52:49.630209 (+ 765us) write_op.cc:270] Start()
0109 20:52:49.630258 (+ 49us) write_op.cc:276] Timestamp: P: 1736455962570998 usec, L: 0
0109 20:52:49.639243 (+ 8985us) write_op.cc:270] Start()
0109 20:52:49.639271 (+ 28us) write_op.cc:276] Timestamp: P: 1736455962572903 usec, L: 0
0109 20:52:49.639470 (+ 199us) write_op.cc:270] Start()
0109 20:52:49.639501 (+ 31us) write_op.cc:276] Timestamp: P: 1736455962573897 usec, L: 0
0109 20:52:49.639630 (+ 129us) write_op.cc:270] Start()
0109 20:52:49.639648 (+ 18us) write_op.cc:276] Timestamp: P: 1736455962574748 usec, L: 0
0109 20:52:49.639786 (+ 138us) write_op.cc:270] Start()
0109 20:52:49.639804 (+ 18us) write_op.cc:276] Timestamp: P: 1736455962575565 usec, L: 0
0109 20:52:49.640165 (+ 361us) write_op.cc:270] Start()
0109 20:52:49.640191 (+ 26us) write_op.cc:276] Timestamp: P: 1736455962661848 usec, L: 0
0109 20:52:49.640347 (+ 156us) write_op.cc:270] Start()
0109 20:52:49.640403 (+ 56us) write_op.cc:276] Timestamp: P: 1736455962663157 usec, L: 0
0109 20:52:49.640534 (+ 131us) write_op.cc:270] Start()
0109 20:52:49.640549 (+ 15us) write_op.cc:276] Timestamp: P: 1736455962663988 usec, L: 0
0109 20:52:49.640661 (+ 112us) write_op.cc:270] Start()
0109 20:52:49.640736 (+ 75us) write_op.cc:276] Timestamp: P: 1736455962674020 usec, L: 0
0109 20:52:49.640868 (+ 132us) write_op.cc:270] Start()
0109 20:52:49.640888 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962674553 usec, L: 0
0109 20:52:49.641033 (+ 145us) write_op.cc:270] Start()
0109 20:52:49.641050 (+ 17us) write_op.cc:276] Timestamp: P: 1736455962675030 usec, L: 0
0109 20:52:49.641159 (+ 109us) write_op.cc:270] Start()
0109 20:52:49.641176 (+ 17us) write_op.cc:276] Timestamp: P: 1736455962675550 usec, L: 0
0109 20:52:49.641293 (+ 117us) write_op.cc:270] Start()
0109 20:52:49.641310 (+ 17us) write_op.cc:276] Timestamp: P: 1736455962676029 usec, L: 0
0109 20:52:49.641431 (+ 121us) write_op.cc:270] Start()
0109 20:52:49.641450 (+ 19us) write_op.cc:276] Timestamp: P: 1736455962676459 usec, L: 0
0109 20:52:49.641571 (+ 121us) write_op.cc:270] Start()
0109 20:52:49.641587 (+ 16us) write_op.cc:276] Timestamp: P: 1736455962676906 usec, L: 0
0109 20:52:49.641689 (+ 102us) write_op.cc:270] Start()
0109 20:52:49.641713 (+ 24us) write_op.cc:276] Timestamp: P: 1736455962677339 usec, L: 0
0109 20:52:49.641836 (+ 123us) write_op.cc:270] Start()
0109 20:52:49.641860 (+ 24us) write_op.cc:276] Timestamp: P: 1736455962677793 usec, L: 0
0109 20:52:49.641976 (+ 116us) write_op.cc:270] Start()
0109 20:52:49.641992 (+ 16us) write_op.cc:276] Timestamp: P: 1736455962685591 usec, L: 0
0109 20:52:49.642109 (+ 117us) write_op.cc:270] Start()
0109 20:52:49.642127 (+ 18us) write_op.cc:276] Timestamp: P: 1736455962686090 usec, L: 0
0109 20:52:49.642259 (+ 132us) write_op.cc:270] Start()
0109 20:52:49.642279 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962707505 usec, L: 0
0109 20:52:49.642449 (+ 170us) write_op.cc:270] Start()
0109 20:52:49.642473 (+ 24us) write_op.cc:276] Timestamp: P: 1736455962709968 usec, L: 0
0109 20:52:49.642622 (+ 149us) write_op.cc:270] Start()
0109 20:52:49.642639 (+ 17us) write_op.cc:276] Timestamp: P: 1736455962711215 usec, L: 0
0109 20:52:49.642774 (+ 135us) write_op.cc:270] Start()
0109 20:52:49.642792 (+ 18us) write_op.cc:276] Timestamp: P: 1736455962767419 usec, L: 0
0109 20:52:49.642919 (+ 127us) write_op.cc:270] Start()
0109 20:52:49.642937 (+ 18us) write_op.cc:276] Timestamp: P: 1736455962791681 usec, L: 0
0109 20:52:49.643068 (+ 131us) write_op.cc:270] Start()
0109 20:52:49.643088 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962792295 usec, L: 0
0109 20:52:49.643217 (+ 129us) write_op.cc:270] Start()
0109 20:52:49.643233 (+ 16us) write_op.cc:276] Timestamp: P: 1736455962831542 usec, L: 0
0109 20:52:49.643351 (+ 118us) write_op.cc:270] Start()
0109 20:52:49.643368 (+ 17us) write_op.cc:276] Timestamp: P: 1736455962843174 usec, L: 0
0109 20:52:49.643469 (+ 101us) write_op.cc:270] Start()
0109 20:52:49.643485 (+ 16us) write_op.cc:276] Timestamp: P: 1736455962851142 usec, L: 0
0109 20:52:49.643607 (+ 122us) write_op.cc:270] Start()
0109 20:52:49.643642 (+ 35us) write_op.cc:276] Timestamp: P: 1736455962857492 usec, L: 0
0109 20:52:49.643795 (+ 153us) write_op.cc:270] Start()
0109 20:52:49.643814 (+ 19us) write_op.cc:276] Timestamp: P: 1736455962877464 usec, L: 0
0109 20:52:49.643953 (+ 139us) write_op.cc:270] Start()
0109 20:52:49.643974 (+ 21us) write_op.cc:276] Timestamp: P: 1736455962895373 usec, L: 0
0109 20:52:49.644106 (+ 132us) write_op.cc:270] Start()
0109 20:52:49.644125 (+ 19us) write_op.cc:276] Timestamp: P: 1736455962896163 usec, L: 0
0109 20:52:49.644249 (+ 124us) write_op.cc:270] Start()
0109 20:52:49.644278 (+ 29us) write_op.cc:276] Timestamp: P: 1736455962897946 usec, L: 0
0109 20:52:49.644411 (+ 133us) write_op.cc:270] Start()
0109 20:52:49.644431 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962905034 usec, L: 0
0109 20:52:49.644557 (+ 126us) write_op.cc:270] Start()
0109 20:52:49.644574 (+ 17us) write_op.cc:276] Timestamp: P: 1736455962910715 usec, L: 0
0109 20:52:49.644737 (+ 163us) write_op.cc:270] Start()
0109 20:52:49.644758 (+ 21us) write_op.cc:276] Timestamp: P: 1736455962919296 usec, L: 0
0109 20:52:49.644973 (+ 215us) write_op.cc:270] Start()
0109 20:52:49.645021 (+ 48us) write_op.cc:276] Timestamp: P: 1736455962920055 usec, L: 0
0109 20:52:49.645153 (+ 132us) write_op.cc:270] Start()
0109 20:52:49.645174 (+ 21us) write_op.cc:276] Timestamp: P: 1736455962920889 usec, L: 0
0109 20:52:49.645301 (+ 127us) write_op.cc:270] Start()
0109 20:52:49.645325 (+ 24us) write_op.cc:276] Timestamp: P: 1736455962922537 usec, L: 0
0109 20:52:49.645459 (+ 134us) write_op.cc:270] Start()
0109 20:52:49.645479 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962926557 usec, L: 0
0109 20:52:49.645606 (+ 127us) write_op.cc:270] Start()
0109 20:52:49.645626 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962930380 usec, L: 0
0109 20:52:49.645765 (+ 139us) write_op.cc:270] Start()
0109 20:52:49.645785 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962931169 usec, L: 0
0109 20:52:49.645922 (+ 137us) write_op.cc:270] Start()
0109 20:52:49.645942 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962953977 usec, L: 0
0109 20:52:49.646056 (+ 114us) write_op.cc:270] Start()
0109 20:52:49.646075 (+ 19us) write_op.cc:276] Timestamp: P: 1736455962960446 usec, L: 0
0109 20:52:49.849534 (+203459us) write_op.cc:270] Start()
0109 20:52:49.849583 (+ 49us) write_op.cc:276] Timestamp: P: 1736455962973700 usec, L: 0
0109 20:52:49.849909 (+ 326us) write_op.cc:270] Start()
0109 20:52:49.849933 (+ 24us) write_op.cc:276] Timestamp: P: 1736455962976340 usec, L: 0
0109 20:52:49.850071 (+ 138us) write_op.cc:270] Start()
0109 20:52:49.850092 (+ 21us) write_op.cc:276] Timestamp: P: 1736455962985344 usec, L: 0
0109 20:52:49.850226 (+ 134us) write_op.cc:270] Start()
0109 20:52:49.850246 (+ 20us) write_op.cc:276] Timestamp: P: 1736455962989714 usec, L: 0
0109 20:52:49.850392 (+ 146us) write_op.cc:270] Start()
0109 20:52:49.850412 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963020506 usec, L: 0
0109 20:52:49.851397 (+ 985us) write_op.cc:270] Start()
0109 20:52:49.851426 (+ 29us) write_op.cc:276] Timestamp: P: 1736455963024517 usec, L: 0
0109 20:52:49.851677 (+ 251us) write_op.cc:270] Start()
0109 20:52:49.851716 (+ 39us) write_op.cc:276] Timestamp: P: 1736455963043469 usec, L: 0
0109 20:52:49.851870 (+ 154us) write_op.cc:270] Start()
0109 20:52:49.851892 (+ 22us) write_op.cc:276] Timestamp: P: 1736455963064864 usec, L: 0
0109 20:52:49.852021 (+ 129us) write_op.cc:270] Start()
0109 20:52:49.852041 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963071774 usec, L: 0
0109 20:52:49.870960 (+ 18919us) write_op.cc:270] Start()
0109 20:52:49.870999 (+ 39us) write_op.cc:276] Timestamp: P: 1736455963137524 usec, L: 0
0109 20:52:49.871210 (+ 211us) write_op.cc:270] Start()
0109 20:52:49.871233 (+ 23us) write_op.cc:276] Timestamp: P: 1736455963138220 usec, L: 0
0109 20:52:49.871362 (+ 129us) write_op.cc:270] Start()
0109 20:52:49.871384 (+ 22us) write_op.cc:276] Timestamp: P: 1736455963144581 usec, L: 0
0109 20:52:49.871507 (+ 123us) write_op.cc:270] Start()
0109 20:52:49.871525 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963151395 usec, L: 0
0109 20:52:49.871642 (+ 117us) write_op.cc:270] Start()
0109 20:52:49.871659 (+ 17us) write_op.cc:276] Timestamp: P: 1736455963162461 usec, L: 0
0109 20:52:49.871792 (+ 133us) write_op.cc:270] Start()
0109 20:52:49.871813 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963187475 usec, L: 0
0109 20:52:49.871944 (+ 131us) write_op.cc:270] Start()
0109 20:52:49.871977 (+ 33us) write_op.cc:276] Timestamp: P: 1736455963197420 usec, L: 0
0109 20:52:49.872095 (+ 118us) write_op.cc:270] Start()
0109 20:52:49.872111 (+ 16us) write_op.cc:276] Timestamp: P: 1736455963197957 usec, L: 0
0109 20:52:49.872224 (+ 113us) write_op.cc:270] Start()
0109 20:52:49.872247 (+ 23us) write_op.cc:276] Timestamp: P: 1736455963198424 usec, L: 0
0109 20:52:49.872378 (+ 131us) write_op.cc:270] Start()
0109 20:52:49.872398 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963198922 usec, L: 0
0109 20:52:49.872585 (+ 187us) write_op.cc:270] Start()
0109 20:52:49.872606 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963199392 usec, L: 0
0109 20:52:49.872795 (+ 189us) write_op.cc:270] Start()
0109 20:52:49.872816 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963200451 usec, L: 0
0109 20:52:49.872944 (+ 128us) write_op.cc:270] Start()
0109 20:52:49.872965 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963200928 usec, L: 0
0109 20:52:49.873091 (+ 126us) write_op.cc:270] Start()
0109 20:52:49.873112 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963201372 usec, L: 0
0109 20:52:49.873241 (+ 129us) write_op.cc:270] Start()
0109 20:52:49.873261 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963210241 usec, L: 0
0109 20:52:49.873379 (+ 118us) write_op.cc:270] Start()
0109 20:52:49.873398 (+ 19us) write_op.cc:276] Timestamp: P: 1736455963212303 usec, L: 0
0109 20:52:49.873539 (+ 141us) write_op.cc:270] Start()
0109 20:52:49.873559 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963220381 usec, L: 0
0109 20:52:49.873694 (+ 135us) write_op.cc:270] Start()
0109 20:52:49.873714 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963230198 usec, L: 0
0109 20:52:49.873888 (+ 174us) write_op.cc:270] Start()
0109 20:52:49.873912 (+ 24us) write_op.cc:276] Timestamp: P: 1736455963282116 usec, L: 0
0109 20:52:49.874086 (+ 174us) write_op.cc:270] Start()
0109 20:52:49.874111 (+ 25us) write_op.cc:276] Timestamp: P: 1736455963282654 usec, L: 0
0109 20:52:49.874303 (+ 192us) write_op.cc:270] Start()
0109 20:52:49.928778 (+ 54475us) write_op.cc:276] Timestamp: P: 1736455963283036 usec, L: 0
0109 20:52:49.929001 (+ 223us) write_op.cc:270] Start()
0109 20:52:49.929021 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963283492 usec, L: 0
0109 20:52:49.929145 (+ 124us) write_op.cc:270] Start()
0109 20:52:49.929161 (+ 16us) write_op.cc:276] Timestamp: P: 1736455963284018 usec, L: 0
0109 20:52:49.929264 (+ 103us) write_op.cc:270] Start()
0109 20:52:49.929278 (+ 14us) write_op.cc:276] Timestamp: P: 1736455963321490 usec, L: 0
0109 20:52:49.929379 (+ 101us) write_op.cc:270] Start()
0109 20:52:49.929394 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963363622 usec, L: 0
0109 20:52:49.929521 (+ 127us) write_op.cc:270] Start()
0109 20:52:49.929554 (+ 33us) write_op.cc:276] Timestamp: P: 1736455963377887 usec, L: 0
0109 20:52:49.929670 (+ 116us) write_op.cc:270] Start()
0109 20:52:49.929683 (+ 13us) write_op.cc:276] Timestamp: P: 1736455963381430 usec, L: 0
0109 20:52:49.929780 (+ 97us) write_op.cc:270] Start()
0109 20:52:49.929795 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963400543 usec, L: 0
0109 20:52:49.929918 (+ 123us) write_op.cc:270] Start()
0109 20:52:49.929946 (+ 28us) write_op.cc:276] Timestamp: P: 1736455963410547 usec, L: 0
0109 20:52:49.930071 (+ 125us) write_op.cc:270] Start()
0109 20:52:49.930091 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963423776 usec, L: 0
0109 20:52:49.930252 (+ 161us) write_op.cc:270] Start()
0109 20:52:49.930273 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963430697 usec, L: 0
0109 20:52:49.930402 (+ 129us) write_op.cc:270] Start()
0109 20:52:49.930421 (+ 19us) write_op.cc:276] Timestamp: P: 1736455963431374 usec, L: 0
0109 20:52:49.930561 (+ 140us) write_op.cc:270] Start()
0109 20:52:49.930576 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963432016 usec, L: 0
0109 20:52:49.930705 (+ 129us) write_op.cc:270] Start()
0109 20:52:49.930723 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963432656 usec, L: 0
0109 20:52:49.930854 (+ 131us) write_op.cc:270] Start()
0109 20:52:49.930872 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963433624 usec, L: 0
0109 20:52:49.930992 (+ 120us) write_op.cc:270] Start()
0109 20:52:49.931007 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963449194 usec, L: 0
0109 20:52:49.931210 (+ 203us) write_op.cc:270] Start()
0109 20:52:49.931229 (+ 19us) write_op.cc:276] Timestamp: P: 1736455963465236 usec, L: 0
0109 20:52:49.931344 (+ 115us) write_op.cc:270] Start()
0109 20:52:49.931360 (+ 16us) write_op.cc:276] Timestamp: P: 1736455963465967 usec, L: 0
0109 20:52:49.931462 (+ 102us) write_op.cc:270] Start()
0109 20:52:49.931478 (+ 16us) write_op.cc:276] Timestamp: P: 1736455963466556 usec, L: 0
0109 20:52:49.931580 (+ 102us) write_op.cc:270] Start()
0109 20:52:49.931595 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963467173 usec, L: 0
0109 20:52:49.931719 (+ 124us) write_op.cc:270] Start()
0109 20:52:49.931745 (+ 26us) write_op.cc:276] Timestamp: P: 1736455963480126 usec, L: 0
0109 20:52:49.931867 (+ 122us) write_op.cc:270] Start()
0109 20:52:49.931887 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963485004 usec, L: 0
0109 20:52:49.932056 (+ 169us) write_op.cc:270] Start()
0109 20:52:49.932078 (+ 22us) write_op.cc:276] Timestamp: P: 1736455963535111 usec, L: 0
0109 20:52:49.932208 (+ 130us) write_op.cc:270] Start()
0109 20:52:49.932225 (+ 17us) write_op.cc:276] Timestamp: P: 1736455963540148 usec, L: 0
0109 20:52:49.945321 (+ 13096us) write_op.cc:270] Start()
0109 20:52:49.945358 (+ 37us) write_op.cc:276] Timestamp: P: 1736455963550947 usec, L: 0
0109 20:52:49.945556 (+ 198us) write_op.cc:270] Start()
0109 20:52:49.945577 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963551449 usec, L: 0
0109 20:52:49.945729 (+ 152us) write_op.cc:270] Start()
0109 20:52:49.945749 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963551852 usec, L: 0
0109 20:52:49.945885 (+ 136us) write_op.cc:270] Start()
0109 20:52:49.945905 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963615335 usec, L: 0
0109 20:52:49.946071 (+ 166us) write_op.cc:270] Start()
0109 20:52:49.946087 (+ 16us) write_op.cc:276] Timestamp: P: 1736455963616920 usec, L: 0
0109 20:52:49.946187 (+ 100us) write_op.cc:270] Start()
0109 20:52:49.946200 (+ 13us) write_op.cc:276] Timestamp: P: 1736455963617948 usec, L: 0
0109 20:52:49.946292 (+ 92us) write_op.cc:270] Start()
0109 20:52:49.946307 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963632620 usec, L: 0
0109 20:52:49.946412 (+ 105us) write_op.cc:270] Start()
0109 20:52:49.946426 (+ 14us) write_op.cc:276] Timestamp: P: 1736455963634451 usec, L: 0
0109 20:52:49.946540 (+ 114us) write_op.cc:270] Start()
0109 20:52:49.946565 (+ 25us) write_op.cc:276] Timestamp: P: 1736455963635277 usec, L: 0
0109 20:52:49.946698 (+ 133us) write_op.cc:270] Start()
0109 20:52:49.946716 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963658933 usec, L: 0
0109 20:52:49.946838 (+ 122us) write_op.cc:270] Start()
0109 20:52:49.946854 (+ 16us) write_op.cc:276] Timestamp: P: 1736455963662436 usec, L: 0
0109 20:52:49.946982 (+ 128us) write_op.cc:270] Start()
0109 20:52:49.947003 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963680165 usec, L: 0
0109 20:52:49.947119 (+ 116us) write_op.cc:270] Start()
0109 20:52:49.947136 (+ 17us) write_op.cc:276] Timestamp: P: 1736455963685538 usec, L: 0
0109 20:52:49.947242 (+ 106us) write_op.cc:270] Start()
0109 20:52:49.947258 (+ 16us) write_op.cc:276] Timestamp: P: 1736455963715890 usec, L: 0
0109 20:52:49.947364 (+ 106us) write_op.cc:270] Start()
0109 20:52:49.947377 (+ 13us) write_op.cc:276] Timestamp: P: 1736455963717304 usec, L: 0
0109 20:52:49.947473 (+ 96us) write_op.cc:270] Start()
0109 20:52:49.947488 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963720865 usec, L: 0
0109 20:52:49.947600 (+ 112us) write_op.cc:270] Start()
0109 20:52:49.947617 (+ 17us) write_op.cc:276] Timestamp: P: 1736455963724222 usec, L: 0
0109 20:52:49.947725 (+ 108us) write_op.cc:270] Start()
0109 20:52:49.947740 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963727328 usec, L: 0
0109 20:52:49.947847 (+ 107us) write_op.cc:270] Start()
0109 20:52:49.947862 (+ 15us) write_op.cc:276] Timestamp: P: 1736455963735475 usec, L: 0
0109 20:52:49.955192 (+ 7330us) write_op.cc:270] Start()
0109 20:52:49.955227 (+ 35us) write_op.cc:276] Timestamp: P: 1736455963744356 usec, L: 0
0109 20:52:49.955423 (+ 196us) write_op.cc:270] Start()
0109 20:52:49.955459 (+ 36us) write_op.cc:276] Timestamp: P: 1736455963745038 usec, L: 0
0109 20:52:49.955599 (+ 140us) write_op.cc:270] Start()
0109 20:52:49.955618 (+ 19us) write_op.cc:276] Timestamp: P: 1736455963745585 usec, L: 0
0109 20:52:49.955747 (+ 129us) write_op.cc:270] Start()
0109 20:52:49.955767 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963767010 usec, L: 0
0109 20:52:49.955889 (+ 122us) write_op.cc:270] Start()
0109 20:52:49.955907 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963779731 usec, L: 0
0109 20:52:49.956050 (+ 143us) write_op.cc:270] Start()
0109 20:52:49.956068 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963794460 usec, L: 0
0109 20:52:49.956191 (+ 123us) write_op.cc:270] Start()
0109 20:52:49.956209 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963799109 usec, L: 0
0109 20:52:49.956336 (+ 127us) write_op.cc:270] Start()
0109 20:52:49.956354 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963802143 usec, L: 0
0109 20:52:49.956492 (+ 138us) write_op.cc:270] Start()
0109 20:52:49.956508 (+ 16us) write_op.cc:276] Timestamp: P: 1736455963873253 usec, L: 0
0109 20:52:49.956657 (+ 149us) write_op.cc:270] Start()
0109 20:52:49.956677 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963875079 usec, L: 0
0109 20:52:49.956841 (+ 164us) write_op.cc:270] Start()
0109 20:52:49.956865 (+ 24us) write_op.cc:276] Timestamp: P: 1736455963889309 usec, L: 0
0109 20:52:49.957020 (+ 155us) write_op.cc:270] Start()
0109 20:52:49.957040 (+ 20us) write_op.cc:276] Timestamp: P: 1736455963893199 usec, L: 0
0109 20:52:49.957192 (+ 152us) write_op.cc:270] Start()
0109 20:52:49.957213 (+ 21us) write_op.cc:276] Timestamp: P: 1736455963902653 usec, L: 0
0109 20:52:49.957367 (+ 154us) write_op.cc:270] Start()
0109 20:52:49.957390 (+ 23us) write_op.cc:276] Timestamp: P: 1736455963911788 usec, L: 0
0109 20:52:49.957521 (+ 131us) write_op.cc:270] Start()
0109 20:52:49.957561 (+ 40us) write_op.cc:276] Timestamp: P: 1736455963931887 usec, L: 0
0109 20:52:49.957686 (+ 125us) write_op.cc:270] Start()
0109 20:52:49.957705 (+ 19us) write_op.cc:276] Timestamp: P: 1736455963943858 usec, L: 0
0109 20:52:49.957843 (+ 138us) write_op.cc:270] Start()
0109 20:52:49.957860 (+ 17us) write_op.cc:276] Timestamp: P: 1736455963945103 usec, L: 0
0109 20:52:49.957986 (+ 126us) write_op.cc:270] Start()
0109 20:52:49.958003 (+ 17us) write_op.cc:276] Timestamp: P: 1736455963956332 usec, L: 0
0109 20:52:49.958135 (+ 132us) write_op.cc:270] Start()
0109 20:52:49.958153 (+ 18us) write_op.cc:276] Timestamp: P: 1736455963968399 usec, L: 0
0109 20:52:49.958299 (+ 146us) write_op.cc:270] Start()
0109 20:52:49.958318 (+ 19us) write_op.cc:276] Timestamp: P: 1736455963972062 usec, L: 0
0109 20:52:49.958517 (+ 199us) write_op.cc:270] Start()
0109 20:52:49.958536 (+ 19us) write_op.cc:276] Timestamp: P: 1736455964011806 usec, L: 0
0109 20:52:49.958649 (+ 113us) write_op.cc:270] Start()
0109 20:52:49.958664 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964017047 usec, L: 0
0109 20:52:49.958757 (+ 93us) write_op.cc:270] Start()
0109 20:52:49.958771 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964034096 usec, L: 0
0109 20:52:49.958862 (+ 91us) write_op.cc:270] Start()
0109 20:52:49.958875 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964036822 usec, L: 0
0109 20:52:49.958973 (+ 98us) write_op.cc:270] Start()
0109 20:52:49.958986 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964039077 usec, L: 0
0109 20:52:49.959072 (+ 86us) write_op.cc:270] Start()
0109 20:52:49.959085 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964045132 usec, L: 0
0109 20:52:49.959172 (+ 87us) write_op.cc:270] Start()
0109 20:52:49.959186 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964053157 usec, L: 0
0109 20:52:49.959282 (+ 96us) write_op.cc:270] Start()
0109 20:52:49.959297 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964060213 usec, L: 0
0109 20:52:49.959384 (+ 87us) write_op.cc:270] Start()
0109 20:52:49.959400 (+ 16us) write_op.cc:276] Timestamp: P: 1736455964113937 usec, L: 0
0109 20:52:49.959492 (+ 92us) write_op.cc:270] Start()
0109 20:52:49.959505 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964138607 usec, L: 0
0109 20:52:49.959591 (+ 86us) write_op.cc:270] Start()
0109 20:52:49.959606 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964237390 usec, L: 0
0109 20:52:49.959703 (+ 97us) write_op.cc:270] Start()
0109 20:52:49.959718 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964244865 usec, L: 0
0109 20:52:49.959834 (+ 116us) write_op.cc:270] Start()
0109 20:52:49.959848 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964248265 usec, L: 0
0109 20:52:49.959964 (+ 116us) write_op.cc:270] Start()
0109 20:52:49.959979 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964270068 usec, L: 0
0109 20:52:49.960096 (+ 117us) write_op.cc:270] Start()
0109 20:52:49.960111 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964331200 usec, L: 0
0109 20:52:49.960211 (+ 100us) write_op.cc:270] Start()
0109 20:52:49.960228 (+ 17us) write_op.cc:276] Timestamp: P: 1736455964333524 usec, L: 0
0109 20:52:49.960332 (+ 104us) write_op.cc:270] Start()
0109 20:52:49.960346 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964344462 usec, L: 0
0109 20:52:49.960458 (+ 112us) write_op.cc:270] Start()
0109 20:52:49.960473 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964353565 usec, L: 0
0109 20:52:49.960597 (+ 124us) write_op.cc:270] Start()
0109 20:52:49.960611 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964354170 usec, L: 0
0109 20:52:49.960707 (+ 96us) write_op.cc:270] Start()
0109 20:52:49.960842 (+ 135us) write_op.cc:276] Timestamp: P: 1736455964354659 usec, L: 0
0109 20:52:49.960954 (+ 112us) write_op.cc:270] Start()
0109 20:52:49.960969 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964355427 usec, L: 0
0109 20:52:49.961067 (+ 98us) write_op.cc:270] Start()
0109 20:52:49.961079 (+ 12us) write_op.cc:276] Timestamp: P: 1736455964355908 usec, L: 0
0109 20:52:49.961168 (+ 89us) write_op.cc:270] Start()
0109 20:52:49.961181 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964356390 usec, L: 0
0109 20:52:49.961277 (+ 96us) write_op.cc:270] Start()
0109 20:52:49.961290 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964365137 usec, L: 0
0109 20:52:49.961387 (+ 97us) write_op.cc:270] Start()
0109 20:52:49.961401 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964366026 usec, L: 0
0109 20:52:49.961494 (+ 93us) write_op.cc:270] Start()
0109 20:52:49.961512 (+ 18us) write_op.cc:276] Timestamp: P: 1736455964366491 usec, L: 0
0109 20:52:49.961604 (+ 92us) write_op.cc:270] Start()
0109 20:52:49.961617 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964431834 usec, L: 0
0109 20:52:49.961706 (+ 89us) write_op.cc:270] Start()
0109 20:52:49.961719 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964569858 usec, L: 0
0109 20:52:49.961810 (+ 91us) write_op.cc:270] Start()
0109 20:52:49.961823 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964577347 usec, L: 0
0109 20:52:49.961913 (+ 90us) write_op.cc:270] Start()
0109 20:52:49.961951 (+ 38us) write_op.cc:276] Timestamp: P: 1736455964577792 usec, L: 0
0109 20:52:49.962069 (+ 118us) write_op.cc:270] Start()
0109 20:52:49.962114 (+ 45us) write_op.cc:276] Timestamp: P: 1736455964578279 usec, L: 0
0109 20:52:49.962259 (+ 145us) write_op.cc:270] Start()
0109 20:52:49.962275 (+ 16us) write_op.cc:276] Timestamp: P: 1736455964578872 usec, L: 0
0109 20:52:49.962364 (+ 89us) write_op.cc:270] Start()
0109 20:52:49.962377 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964597500 usec, L: 0
0109 20:52:49.962466 (+ 89us) write_op.cc:270] Start()
0109 20:52:49.962480 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964599375 usec, L: 0
0109 20:52:49.962575 (+ 95us) write_op.cc:270] Start()
0109 20:52:49.962588 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964606763 usec, L: 0
0109 20:52:49.962680 (+ 92us) write_op.cc:270] Start()
0109 20:52:49.962694 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964607318 usec, L: 0
0109 20:52:49.962806 (+ 112us) write_op.cc:270] Start()
0109 20:52:49.962821 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964621445 usec, L: 0
0109 20:52:49.962919 (+ 98us) write_op.cc:270] Start()
0109 20:52:49.962942 (+ 23us) write_op.cc:276] Timestamp: P: 1736455964622018 usec, L: 0
0109 20:52:49.963051 (+ 109us) write_op.cc:270] Start()
0109 20:52:49.963084 (+ 33us) write_op.cc:276] Timestamp: P: 1736455964626013 usec, L: 0
0109 20:52:49.963212 (+ 128us) write_op.cc:270] Start()
0109 20:52:49.963228 (+ 16us) write_op.cc:276] Timestamp: P: 1736455964629606 usec, L: 0
0109 20:52:49.963374 (+ 146us) write_op.cc:270] Start()
0109 20:52:49.963390 (+ 16us) write_op.cc:276] Timestamp: P: 1736455964640054 usec, L: 0
0109 20:52:49.963492 (+ 102us) write_op.cc:270] Start()
0109 20:52:49.963506 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964642277 usec, L: 0
0109 20:52:49.963599 (+ 93us) write_op.cc:270] Start()
0109 20:52:49.963612 (+ 13us) write_op.cc:276] Timestamp: P: 1736455964643452 usec, L: 0
0109 20:52:49.963705 (+ 93us) write_op.cc:270] Start()
0109 20:52:49.963719 (+ 14us) write_op.cc:276] Timestamp: P: 1736455964655316 usec, L: 0
0109 20:52:49.963846 (+ 127us) write_op.cc:270] Start()
0109 20:52:49.963861 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964658159 usec, L: 0
0109 20:52:49.963962 (+ 101us) write_op.cc:270] Start()
0109 20:52:49.963977 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964673280 usec, L: 0
0109 20:52:49.964085 (+ 108us) write_op.cc:270] Start()
0109 20:52:49.964100 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964707452 usec, L: 0
0109 20:52:49.964203 (+ 103us) write_op.cc:270] Start()
0109 20:52:49.964220 (+ 17us) write_op.cc:276] Timestamp: P: 1736455964800210 usec, L: 0
0109 20:52:49.964324 (+ 104us) write_op.cc:270] Start()
0109 20:52:49.964343 (+ 19us) write_op.cc:276] Timestamp: P: 1736455964802314 usec, L: 0
0109 20:52:49.964449 (+ 106us) write_op.cc:270] Start()
0109 20:52:49.964464 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964806657 usec, L: 0
0109 20:52:49.964562 (+ 98us) write_op.cc:270] Start()
0109 20:52:49.964577 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964815816 usec, L: 0
0109 20:52:49.964679 (+ 102us) write_op.cc:270] Start()
0109 20:52:49.964694 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964817527 usec, L: 0
0109 20:52:49.964857 (+ 163us) write_op.cc:270] Start()
0109 20:52:49.964878 (+ 21us) write_op.cc:276] Timestamp: P: 1736455964827084 usec, L: 0
0109 20:52:49.965046 (+ 168us) write_op.cc:270] Start()
0109 20:52:49.965064 (+ 18us) write_op.cc:276] Timestamp: P: 1736455964830572 usec, L: 0
0109 20:52:49.965192 (+ 128us) write_op.cc:270] Start()
0109 20:52:49.965211 (+ 19us) write_op.cc:276] Timestamp: P: 1736455964831070 usec, L: 0
0109 20:52:49.965659 (+ 448us) write_op.cc:270] Start()
0109 20:52:49.965682 (+ 23us) write_op.cc:276] Timestamp: P: 1736455964831692 usec, L: 0
0109 20:52:49.965807 (+ 125us) write_op.cc:270] Start()
0109 20:52:49.965823 (+ 16us) write_op.cc:276] Timestamp: P: 1736455964839375 usec, L: 0
0109 20:52:49.965956 (+ 133us) write_op.cc:270] Start()
0109 20:52:49.965972 (+ 16us) write_op.cc:276] Timestamp: P: 1736455964839887 usec, L: 0
0109 20:52:49.966133 (+ 161us) write_op.cc:270] Start()
0109 20:52:49.966148 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964861231 usec, L: 0
0109 20:52:49.966257 (+ 109us) write_op.cc:270] Start()
0109 20:52:49.966272 (+ 15us) write_op.cc:276] Timestamp: P: 1736455964861760 usec, L: 0
0109 20:52:49.966415 (+ 143us) write_op.cc:270] Start()
0109 20:52:49.966437 (+ 22us) write_op.cc:276] Timestamp: P: 17364559
I20250109 20:52:50.764125 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4f90c364ce844c9ea8eccb4ae23f8dce
I20250109 20:52:50.764127 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a9910f88574eac90db9e00138c8dda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.765139 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e16a1d9a5b374b8dbf725eb32ffadb9e
I20250109 20:52:50.765399 1166 log.cc:1198] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/46a9910f88574eac90db9e00138c8dda
I20250109 20:52:50.765818 1166 ts_tablet_manager.cc:1939] T 46a9910f88574eac90db9e00138c8dda P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.766160 1160 tablet_replica.cc:331] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:50.766351 1160 raft_consensus.cc:2238] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:50.766599 1160 pending_rounds.cc:62] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Trying to abort 1 pending ops.
I20250109 20:52:50.766664 1160 pending_rounds.cc:69] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523655708815360 op_type: NO_OP noop_request { }
I20250109 20:52:50.766862 1160 raft_consensus.cc:2883] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:50.766960 1160 raft_consensus.cc:2267] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.767266 1176 tablet_bootstrap.cc:492] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.767478 1176 ts_tablet_manager.cc:1397] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.003s
I20250109 20:52:50.768159 1176 raft_consensus.cc:357] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.768390 1176 raft_consensus.cc:383] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.768451 1176 raft_consensus.cc:738] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.768703 1176 consensus_queue.cc:260] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.769419 1176 ts_tablet_manager.cc:1428] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:50.768682 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.769851 1176 tablet_bootstrap.cc:492] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.771868 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_986f5a94df1245f7930aca6efd8614a2" } modify_external_catalogs: true
I20250109 20:52:50.772022 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_986f5a94df1245f7930aca6efd8614a2" } modify_external_catalogs: true
I20250109 20:52:50.772740 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 46a9910f88574eac90db9e00138c8dda (table default.Table_9d14aaae5e2d40a3842ad98fabf33015 [id=1e8fda0dc6034fdf9aa9b52712005046]) successfully deleted
I20250109 20:52:50.772935 1176 tablet_bootstrap.cc:654] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.773512 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_171f660e57e54000a542c3e8bcd0bfb5" } new_table_name: "default.Table_634a6ecf6a7b4731ae4ac6e03c5375d1" modify_external_catalogs: true
I20250109 20:52:50.773994 758 tablet_service.cc:1514] Processing DeleteTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.775296 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e75baed496db483c91ad7464f1678e0d" } new_table_name: "default.Table_9849e2509cde44b7ad7db247abe62a79" modify_external_catalogs: true
W20250109 20:52:50.775403 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 35a39fac67894a208dd7ec3153b05e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a39fac67894a208dd7ec3153b05e4a already in progress: creating tablet
W20250109 20:52:50.775640 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b405d16cc62444f2ad94fb69bb9fb6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b405d16cc62444f2ad94fb69bb9fb6ee already in progress: creating tablet
W20250109 20:52:50.775817 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ea16fcc30f2c4b2eac53f20fcebd043f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea16fcc30f2c4b2eac53f20fcebd043f already in progress: creating tablet
W20250109 20:52:50.776013 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9d7587432d7345e4a225fc7d5c968c27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d7587432d7345e4a225fc7d5c968c27 already in progress: creating tablet
W20250109 20:52:50.776206 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 829bec4f66c24180afd5271057a7d0e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829bec4f66c24180afd5271057a7d0e8 already in progress: creating tablet
W20250109 20:52:50.776512 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 28cd9cd48910497ba3e4b159583cb7a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28cd9cd48910497ba3e4b159583cb7a6 already in progress: creating tablet
I20250109 20:52:50.777719 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9c4c55aacaeb4e3faa678a177e936bbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.778151 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c4c55aacaeb4e3faa678a177e936bbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.778817 1166 tablet_replica.cc:331] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:50.778997 1166 raft_consensus.cc:2238] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.779132 1166 raft_consensus.cc:2267] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.779598 1160 ts_tablet_manager.cc:1905] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.785684 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4ffd6adbaf4f7083e7ba1a518ed813 (DEFAULT_TABLE table=default.Table_69a70cbdfe8148838135d7f7da4e6619 [id=64752eaca1894fe2a1fe46af9a52337a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.785956 756 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4ffd6adbaf4f7083e7ba1a518ed813 (DEFAULT_TABLE table=default.Table_69a70cbdfe8148838135d7f7da4e6619 [id=64752eaca1894fe2a1fe46af9a52337a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.786237 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4ffd6adbaf4f7083e7ba1a518ed813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.786309 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4ffd6adbaf4f7083e7ba1a518ed813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.789027 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f90c364ce844c9ea8eccb4ae23f8dce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.789336 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f90c364ce844c9ea8eccb4ae23f8dce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.789530 1157 tablet_replica.cc:331] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:50.789434 744 tablet_service.cc:1467] Processing CreateTablet for tablet 37ee437300414eccba9926e824f72f27 (DEFAULT_TABLE table=default.Table_7f957a53ae794811bf294551bf3f6e00 [id=8097eb217f1844ec8e2dc236b2f19256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.789836 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ee437300414eccba9926e824f72f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.790495 748 tablet_service.cc:1467] Processing CreateTablet for tablet 678ab724db1a42e2b23bbe6cd7652db7 (DEFAULT_TABLE table=default.Table_546faac34e4d4250b0845e4886d01377 [id=db91cf220f2d4191832b2c3c6cddf99d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.790915 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678ab724db1a42e2b23bbe6cd7652db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.791783 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_13f472ba51244613ab6c1c0cc5e9f9e4" } modify_external_catalogs: true
I20250109 20:52:50.791939 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_13f472ba51244613ab6c1c0cc5e9f9e4" } modify_external_catalogs: true
I20250109 20:52:50.776458 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cf3f4f1f920642a396c170041d629601"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.793192 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf3f4f1f920642a396c170041d629601 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.796101 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_47a796218c0848d9990864cc20f7ae05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.796242 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a45cd5ffe7040e385d8cdfb97ed011a consensus state has no leader
W20250109 20:52:50.796679 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47a796218c0848d9990864cc20f7ae05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.796754 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_72bdbf7541534c67bc7d545c15638587" } new_table_name: "default.Table_2c68d926e1b8418aa0339da1c4cb17cf" modify_external_catalogs: true
I20250109 20:52:50.796236 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e0cf5166142546989eb825944cae0691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.797940 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0cf5166142546989eb825944cae0691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.798269 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0e82dbd939465ca7044f0aa1f46f06 (DEFAULT_TABLE table=default.Table_3dbeca6078054ff0877acae499fb525c [id=b70cbd7bddf9479c83c7def3f4b819cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.798637 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0e82dbd939465ca7044f0aa1f46f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.799039 739 tablet_service.cc:1467] Processing CreateTablet for tablet faeec3dbcaff4de98e94357f20e7e8ad (DEFAULT_TABLE table=default.Table_472ddec2dea04b1d83cb147944a488c9 [id=2da6a2c99a6e40d687307f107bef5264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.799404 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faeec3dbcaff4de98e94357f20e7e8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.805454 1157 raft_consensus.cc:2238] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.805639 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c865faed94c3493ea638826bdcd4253d
I20250109 20:52:50.805603 1026 tablet_service.cc:1467] Processing CreateTablet for tablet aa89498a0e554c6b8f69f0c0e407da32 (DEFAULT_TABLE table=default.Table_da940b25eb974021bef8e7d3aa57dfc0 [id=6d8ef8d222f743129c8160463e937f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.805681 1157 raft_consensus.cc:2267] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.806162 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b50c91807734e86840e874886dc0749 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.806433 752 tablet_service.cc:1514] Processing DeleteTablet for tablet c865faed94c3493ea638826bdcd4253d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.806658 1157 ts_tablet_manager.cc:1905] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.807214 744 tablet_service.cc:1467] Processing CreateTablet for tablet c5d455aef15046e793f97f0ba2c994ed (DEFAULT_TABLE table=default.Table_897456425a2b4e48949409b18f6d770e [id=2e1baf4c6fe54484affbf1efdc176b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.807307 754 tablet_service.cc:1467] Processing CreateTablet for tablet 9629fe30752f49478afad9b93dc235eb (DEFAULT_TABLE table=default.Table_3bc3ade0c5964f94802bd73bd9e69f21 [id=fcb76535fbbe44e29ef627a3ae562ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.807674 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9629fe30752f49478afad9b93dc235eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.807785 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa89498a0e554c6b8f69f0c0e407da32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.807695 739 tablet_service.cc:1467] Processing CreateTablet for tablet d4c0f80ec2094da0893fb0a92223b1cd (DEFAULT_TABLE table=default.Table_dcb1259a70b64057b0090a78234d47a6 [id=80c904806c7444329fbca174ae613fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.810106 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c0f80ec2094da0893fb0a92223b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.810164 758 tablet_service.cc:1467] Processing CreateTablet for tablet 049321d28ab241138247dcefa0411f54 (DEFAULT_TABLE table=default.Table_e79e938de6ba4b929f53a7f9656e8416 [id=73d86ca9226d4a84aacc05835ada9e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.810674 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049321d28ab241138247dcefa0411f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.813767 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet fce062dc0ac144a19f8fa3eef4a099b4
I20250109 20:52:50.815398 1029 tablet_service.cc:1467] Processing CreateTablet for tablet c5d455aef15046e793f97f0ba2c994ed (DEFAULT_TABLE table=default.Table_897456425a2b4e48949409b18f6d770e [id=2e1baf4c6fe54484affbf1efdc176b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.815984 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 5757b98100804c688f67e7111db2b622 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.816393 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 049321d28ab241138247dcefa0411f54 (DEFAULT_TABLE table=default.Table_e79e938de6ba4b929f53a7f9656e8416 [id=73d86ca9226d4a84aacc05835ada9e03]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:50.816982 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b56d2c0706c84195acceb97ef4985c14 consensus state has no leader
W20250109 20:52:50.818308 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d57a8bbb3da84088a98c816102e4e9df consensus state has no leader
I20250109 20:52:50.822754 453 master_service.cc:946] ReplaceTablet: received request to replace tablet dc235614171f4a61b75492ddeb3f0fe6 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.813675 749 tablet_service.cc:1467] Processing CreateTablet for tablet aa89498a0e554c6b8f69f0c0e407da32 (DEFAULT_TABLE table=default.Table_da940b25eb974021bef8e7d3aa57dfc0 [id=6d8ef8d222f743129c8160463e937f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.823899 742 tablet_service.cc:1467] Processing CreateTablet for tablet d9f18f244a344272bba2bc085686b61d (DEFAULT_TABLE table=default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.828819 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 37ee437300414eccba9926e824f72f27 (DEFAULT_TABLE table=default.Table_7f957a53ae794811bf294551bf3f6e00 [id=8097eb217f1844ec8e2dc236b2f19256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.829450 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9a67ee66b9a049a78f9a0788ec87dd26" } new_table_name: "default.Table_09ed969c8f9a442392025cc69ce2ea70" modify_external_catalogs: true
I20250109 20:52:50.829478 1035 tablet_service.cc:1467] Processing CreateTablet for tablet faeec3dbcaff4de98e94357f20e7e8ad (DEFAULT_TABLE table=default.Table_472ddec2dea04b1d83cb147944a488c9 [id=2da6a2c99a6e40d687307f107bef5264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.830421 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ddedf78c0442daa9b1d05334f12b4d with delete_type TABLET_DATA_DELETED (replaced by tablet 7b7dc6809d6547f6b0354b64d7149bf4) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.831467 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 678ab724db1a42e2b23bbe6cd7652db7 (DEFAULT_TABLE table=default.Table_546faac34e4d4250b0845e4886d01377 [id=db91cf220f2d4191832b2c3c6cddf99d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.831517 1166 ts_tablet_manager.cc:1905] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.832019 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d455aef15046e793f97f0ba2c994ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.832970 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.833343 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de7acfa7f941caa94bdbb888cf841e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.833779 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet c865faed94c3493ea638826bdcd4253d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.833812 1028 tablet_service.cc:1467] Processing CreateTablet for tablet d4c0f80ec2094da0893fb0a92223b1cd (DEFAULT_TABLE table=default.Table_dcb1259a70b64057b0090a78234d47a6 [id=80c904806c7444329fbca174ae613fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.833967 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0e82dbd939465ca7044f0aa1f46f06 (DEFAULT_TABLE table=default.Table_3dbeca6078054ff0877acae499fb525c [id=b70cbd7bddf9479c83c7def3f4b819cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.834265 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 9629fe30752f49478afad9b93dc235eb (DEFAULT_TABLE table=default.Table_3bc3ade0c5964f94802bd73bd9e69f21 [id=fcb76535fbbe44e29ef627a3ae562ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.834838 1176 tablet_bootstrap.cc:492] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.835019 1176 ts_tablet_manager.cc:1397] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.003s
I20250109 20:52:50.836050 1176 raft_consensus.cc:357] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.836263 1176 raft_consensus.cc:383] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.836616 1176 raft_consensus.cc:738] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.836879 1176 consensus_queue.cc:260] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.837304 1502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:50.837584 1502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: LEADER
I20250109 20:52:50.839895 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e1503bbd879343928760e2b8a1192c3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.840412 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1503bbd879343928760e2b8a1192c3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.842024 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f90c364ce844c9ea8eccb4ae23f8dce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.842165 895 tablet_service.cc:1467] Processing CreateTablet for tablet 049321d28ab241138247dcefa0411f54 (DEFAULT_TABLE table=default.Table_e79e938de6ba4b929f53a7f9656e8416 [id=73d86ca9226d4a84aacc05835ada9e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.842672 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049321d28ab241138247dcefa0411f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.843044 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c0f80ec2094da0893fb0a92223b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.845227 1176 ts_tablet_manager.cc:1428] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:52:50.845418 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.845494 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ee437300414eccba9926e824f72f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.845538 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d9f18f244a344272bba2bc085686b61d (DEFAULT_TABLE table=default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.846532 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049321d28ab241138247dcefa0411f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.847599 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faeec3dbcaff4de98e94357f20e7e8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.854107 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9629fe30752f49478afad9b93dc235eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.854529 885 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4ffd6adbaf4f7083e7ba1a518ed813 (DEFAULT_TABLE table=default.Table_69a70cbdfe8148838135d7f7da4e6619 [id=64752eaca1894fe2a1fe46af9a52337a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.854903 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f18f244a344272bba2bc085686b61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.855085 898 tablet_service.cc:1467] Processing CreateTablet for tablet 37ee437300414eccba9926e824f72f27 (DEFAULT_TABLE table=default.Table_7f957a53ae794811bf294551bf3f6e00 [id=8097eb217f1844ec8e2dc236b2f19256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.855558 897 tablet_service.cc:1467] Processing CreateTablet for tablet 678ab724db1a42e2b23bbe6cd7652db7 (DEFAULT_TABLE table=default.Table_546faac34e4d4250b0845e4886d01377 [id=db91cf220f2d4191832b2c3c6cddf99d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.856073 893 tablet_service.cc:1467] Processing CreateTablet for tablet aa89498a0e554c6b8f69f0c0e407da32 (DEFAULT_TABLE table=default.Table_da940b25eb974021bef8e7d3aa57dfc0 [id=6d8ef8d222f743129c8160463e937f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.856560 890 tablet_service.cc:1467] Processing CreateTablet for tablet c5d455aef15046e793f97f0ba2c994ed (DEFAULT_TABLE table=default.Table_897456425a2b4e48949409b18f6d770e [id=2e1baf4c6fe54484affbf1efdc176b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.856829 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678ab724db1a42e2b23bbe6cd7652db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.857141 888 tablet_service.cc:1467] Processing CreateTablet for tablet faeec3dbcaff4de98e94357f20e7e8ad (DEFAULT_TABLE table=default.Table_472ddec2dea04b1d83cb147944a488c9 [id=2da6a2c99a6e40d687307f107bef5264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.857710 896 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0e82dbd939465ca7044f0aa1f46f06 (DEFAULT_TABLE table=default.Table_3dbeca6078054ff0877acae499fb525c [id=b70cbd7bddf9479c83c7def3f4b819cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.858265 880 tablet_service.cc:1467] Processing CreateTablet for tablet d4c0f80ec2094da0893fb0a92223b1cd (DEFAULT_TABLE table=default.Table_dcb1259a70b64057b0090a78234d47a6 [id=80c904806c7444329fbca174ae613fc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.858860 883 tablet_service.cc:1467] Processing CreateTablet for tablet 9629fe30752f49478afad9b93dc235eb (DEFAULT_TABLE table=default.Table_3bc3ade0c5964f94802bd73bd9e69f21 [id=fcb76535fbbe44e29ef627a3ae562ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.859262 881 tablet_service.cc:1514] Processing DeleteTablet for tablet c865faed94c3493ea638826bdcd4253d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.859500 882 tablet_service.cc:1514] Processing DeleteTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.859848 887 tablet_service.cc:1467] Processing CreateTablet for tablet d9f18f244a344272bba2bc085686b61d (DEFAULT_TABLE table=default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:50.861392 1165 tablet_bootstrap.cc:492] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:50.861579 1165 ts_tablet_manager.cc:1397] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.223s user 0.017s sys 0.031s
I20250109 20:52:50.862623 1165 raft_consensus.cc:357] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.862908 1165 raft_consensus.cc:383] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.863040 1165 raft_consensus.cc:738] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.863397 1165 consensus_queue.cc:260] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.864826 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.864923 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0e82dbd939465ca7044f0aa1f46f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.865442 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c4ffd6adbaf4f7083e7ba1a518ed813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.867465 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f18f244a344272bba2bc085686b61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.869218 1671 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:52:50.869657 1671 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:50.871150 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 530 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:52:50.873432 1671 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
I20250109 20:52:50.875396 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ee437300414eccba9926e824f72f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.876765 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d455aef15046e793f97f0ba2c994ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.879169 1176 tablet_bootstrap.cc:492] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.880774 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faeec3dbcaff4de98e94357f20e7e8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.883715 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 530 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:52:50.884151 1165 ts_tablet_manager.cc:1428] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250109 20:52:50.884364 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c0f80ec2094da0893fb0a92223b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.885668 1165 tablet_bootstrap.cc:492] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:50.887266 1165 tablet_bootstrap.cc:654] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.884399 1400 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
I20250109 20:52:50.888649 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa89498a0e554c6b8f69f0c0e407da32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.892889 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0e82dbd939465ca7044f0aa1f46f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.893463 1176 tablet_bootstrap.cc:654] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.893452 1671 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:52:50.896957 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9629fe30752f49478afad9b93dc235eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.900941 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678ab724db1a42e2b23bbe6cd7652db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.902892 1165 tablet_bootstrap.cc:492] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:50.903084 1165 ts_tablet_manager.cc:1397] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.005s
I20250109 20:52:50.903901 1165 raft_consensus.cc:357] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.904194 1165 raft_consensus.cc:383] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.904320 1165 raft_consensus.cc:738] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.904667 1165 consensus_queue.cc:260] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.907433 1165 ts_tablet_manager.cc:1428] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:50.908666 1165 tablet_bootstrap.cc:492] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:50.910043 1590 raft_consensus.cc:491] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.910226 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f18f244a344272bba2bc085686b61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.911535 754 tablet_service.cc:1514] Processing DeleteTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.913187 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d455aef15046e793f97f0ba2c994ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.913242 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 3d8d67b624984a0bb327d045c55476da from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.913699 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa89498a0e554c6b8f69f0c0e407da32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:50.913795 1451 raft_consensus.cc:491] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:50.913983 1451 raft_consensus.cc:513] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.914685 1451 leader_election.cc:290] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:50.915392 1590 raft_consensus.cc:513] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:50.916821 1158 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.038s sys 0.063s
I20250109 20:52:50.919168 1590 leader_election.cc:290] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:50.923643 1166 ts_tablet_manager.cc:1918] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.924546 1176 tablet_bootstrap.cc:492] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.924849 1176 ts_tablet_manager.cc:1397] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.078s user 0.001s sys 0.004s
I20250109 20:52:50.922427 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4f256d59a2459e95be0f950a5a86eb" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:50.925359 990 leader_election.cc:343] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:50.925791 1176 raft_consensus.cc:357] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.926013 1176 raft_consensus.cc:383] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.926388 1176 raft_consensus.cc:738] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.926643 1176 consensus_queue.cc:260] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.928345 1176 ts_tablet_manager.cc:1428] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:50.928695 1176 tablet_bootstrap.cc:492] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:50.930560 1176 tablet_bootstrap.cc:654] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.931720 1166 log.cc:1198] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6716e7428ce240fca7ca5b02f564a5df
I20250109 20:52:50.932122 1166 ts_tablet_manager.cc:1939] T 6716e7428ce240fca7ca5b02f564a5df P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.933800 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6716e7428ce240fca7ca5b02f564a5df (table default.Table_bb480bfd701049cc8b814e0fe8b4c9ce [id=49d5df19d5124b71993545f365062601]) successfully deleted
I20250109 20:52:50.935377 1166 tablet_replica.cc:331] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:50.935622 1166 raft_consensus.cc:2238] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.935849 1166 raft_consensus.cc:2267] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.936421 1166 ts_tablet_manager.cc:1905] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.943336 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4f256d59a2459e95be0f950a5a86eb" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:50.949262 1176 tablet_bootstrap.cc:492] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:50.949422 1176 ts_tablet_manager.cc:1397] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.004s
I20250109 20:52:50.950102 1157 ts_tablet_manager.cc:1918] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.952046 1166 ts_tablet_manager.cc:1918] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:50.952037 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet dc235614171f4a61b75492ddeb3f0fe6
I20250109 20:52:50.952154 1166 log.cc:1198] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c03524194a7b4e34a20424091e308ac4
I20250109 20:52:50.952487 1166 ts_tablet_manager.cc:1939] T c03524194a7b4e34a20424091e308ac4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:50.952832 453 catalog_manager.cc:6596] ReplaceTablet: tablet dc235614171f4a61b75492ddeb3f0fe6 deleted and replaced by tablet 67be9533983443cbb8fc52031216a973
I20250109 20:52:50.953092 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet dc235614171f4a61b75492ddeb3f0fe6 with delete_type TABLET_DATA_DELETED (replaced by tablet 67be9533983443cbb8fc52031216a973) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:50.953697 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3293f49190684b72a5c41031e621bd36 consensus state has no leader
I20250109 20:52:50.953581 1176 raft_consensus.cc:357] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.953796 1176 raft_consensus.cc:383] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:50.953877 1176 raft_consensus.cc:738] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:50.954121 1176 consensus_queue.cc:260] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.954646 1176 ts_tablet_manager.cc:1428] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:50.954653 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 59e1e8754625463ea42a92ef65a89214 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:50.955063 1176 tablet_bootstrap.cc:492] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:50.955539 713 leader_election.cc:343] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 54d11b9148dc42bc91348e837af73972
I20250109 20:52:50.956276 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d08a3af516364157a07af7368a5f95f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.957229 1166 tablet_replica.cc:331] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:50.957404 1166 raft_consensus.cc:2238] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.957552 1166 raft_consensus.cc:2267] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.958473 1176 tablet_bootstrap.cc:654] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:50.958739 712 leader_election.cc:343] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Not found: Tablet not found: 54d11b9148dc42bc91348e837af73972
I20250109 20:52:50.958810 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet c03524194a7b4e34a20424091e308ac4 (table default.Table_df185d0c630644438a9e8cc5fe14446b [id=d89b416c6f8d4417a6ccb26b443b2ef4]) successfully deleted
I20250109 20:52:50.958897 712 leader_election.cc:304] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:50.961328 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4160bb4ba40347bda48f8fa88f8b6001
I20250109 20:52:50.961582 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bc4cb5b8559b475fb49723c73456ce40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.962190 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc4cb5b8559b475fb49723c73456ce40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:50.963423 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1b50c91807734e86840e874886dc0749
I20250109 20:52:50.963871 1157 log.cc:1198] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/607a78e281594e9a92c6aa63ee7bdf8d
I20250109 20:52:50.963910 449 catalog_manager.cc:6596] ReplaceTablet: tablet 1b50c91807734e86840e874886dc0749 deleted and replaced by tablet 39f0cc2be25940d08fad42a8c8d8a07b
I20250109 20:52:50.955415 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6dfedfbc9bba4dc8a4fb00f68b3ba689"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:50.964381 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e16a1d9a5b374b8dbf725eb32ffadb9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e16a1d9a5b374b8dbf725eb32ffadb9e already in progress: creating tablet
W20250109 20:52:50.964599 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dfedfbc9bba4dc8a4fb00f68b3ba689 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:50.964884 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4f90c364ce844c9ea8eccb4ae23f8dce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f90c364ce844c9ea8eccb4ae23f8dce already in progress: creating tablet
W20250109 20:52:50.965127 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet fce062dc0ac144a19f8fa3eef4a099b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fce062dc0ac144a19f8fa3eef4a099b4 already in progress: creating tablet
I20250109 20:52:50.966078 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b50c91807734e86840e874886dc0749 with delete_type TABLET_DATA_DELETED (replaced by tablet 39f0cc2be25940d08fad42a8c8d8a07b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:50.966568 1157 ts_tablet_manager.cc:1939] T 607a78e281594e9a92c6aa63ee7bdf8d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:50.967191 742 tablet_service.cc:1514] Processing DeleteTablet for tablet dc235614171f4a61b75492ddeb3f0fe6 with delete_type TABLET_DATA_DELETED (replaced by tablet 67be9533983443cbb8fc52031216a973) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.967512 918 raft_consensus.cc:2463] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
W20250109 20:52:50.968778 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0538f4d3608a454b9dd21a0fa42068a0 consensus state has no leader
I20250109 20:52:50.968828 1160 ts_tablet_manager.cc:1918] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:50.968981 1160 log.cc:1198] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/46a9910f88574eac90db9e00138c8dda
I20250109 20:52:50.969415 1160 ts_tablet_manager.cc:1939] T 46a9910f88574eac90db9e00138c8dda P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:50.970252 1160 tablet_replica.cc:331] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:50.970413 1160 raft_consensus.cc:2238] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.970943 1157 tablet_replica.cc:331] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:50.971138 1157 raft_consensus.cc:2238] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:50.971310 1157 raft_consensus.cc:2267] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.971987 1157 ts_tablet_manager.cc:1905] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.972599 989 leader_election.cc:304] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f
I20250109 20:52:50.973129 1451 raft_consensus.cc:2798] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:50.973240 1451 raft_consensus.cc:491] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:50.973275 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.973351 1451 raft_consensus.cc:3054] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:52:50.973951 1158 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.531s user 0.038s sys 0.063s
I20250109 20:52:50.974766 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b50c91807734e86840e874886dc0749 with delete_type TABLET_DATA_DELETED (replaced by tablet 39f0cc2be25940d08fad42a8c8d8a07b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:50.978845 1681 tablet.cc:1721] T da2d6dda69da4a59b055106c1041ec2b P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:50.975847 1158 tablet_bootstrap.cc:492] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:50.984701 1166 ts_tablet_manager.cc:1905] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.992264 1451 raft_consensus.cc:513] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:50.970559 1160 raft_consensus.cc:2267] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:50.973489 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b50c91807734e86840e874886dc0749 with delete_type TABLET_DATA_DELETED (replaced by tablet 39f0cc2be25940d08fad42a8c8d8a07b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.986706 1165 tablet_bootstrap.cc:654] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:50.987146 897 tablet_service.cc:1514] Processing DeleteTablet for tablet dc235614171f4a61b75492ddeb3f0fe6 with delete_type TABLET_DATA_DELETED (replaced by tablet 67be9533983443cbb8fc52031216a973) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:50.994989 1160 ts_tablet_manager.cc:1905] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:50.998327 1451 leader_election.cc:290] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:50.998669 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4f256d59a2459e95be0f950a5a86eb" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:50.998939 918 raft_consensus.cc:3054] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:51.006327 918 raft_consensus.cc:2463] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:51.007004 989 leader_election.cc:304] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:51.009508 1451 raft_consensus.cc:2798] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:51.009775 1451 raft_consensus.cc:695] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:51.010186 1451 consensus_queue.cc:237] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.010504 1158 ts_tablet_manager.cc:1397] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.601s user 0.040s sys 0.064s
I20250109 20:52:51.010535 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4f256d59a2459e95be0f950a5a86eb" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
W20250109 20:52:51.011222 990 leader_election.cc:343] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.011463 1158 raft_consensus.cc:357] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.011651 1158 raft_consensus.cc:383] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.011722 1158 raft_consensus.cc:738] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.012167 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 607a78e281594e9a92c6aa63ee7bdf8d (table default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]) successfully deleted
W20250109 20:52:51.012367 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 44037.9us
I20250109 20:52:51.013764 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:51.014104 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 72a6cf525cb44c428f7a39a25c4e62dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a6cf525cb44c428f7a39a25c4e62dc already in progress: creating tablet
I20250109 20:52:51.015362 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.015698 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3d8d67b624984a0bb327d045c55476da
I20250109 20:52:51.016021 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 46a9910f88574eac90db9e00138c8dda (table default.Table_9d14aaae5e2d40a3842ad98fabf33015 [id=1e8fda0dc6034fdf9aa9b52712005046]) successfully deleted
I20250109 20:52:51.011945 1158 consensus_queue.cc:260] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.017243 1158 ts_tablet_manager.cc:1428] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250109 20:52:51.017473 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 58747.2us
I20250109 20:52:51.018154 1683 tablet.cc:1721] T da2d6dda69da4a59b055106c1041ec2b P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:51.020359 1158 tablet_bootstrap.cc:492] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:51.023873 1158 tablet_bootstrap.cc:654] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.028249 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f33bf5f9431146a5b111b0448a62e85d" } modify_external_catalogs: true
I20250109 20:52:51.028510 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f33bf5f9431146a5b111b0448a62e85d" } modify_external_catalogs: true
I20250109 20:52:51.029413 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2bdba403f4a24ce392324eeb74d3ed0e" } modify_external_catalogs: true
I20250109 20:52:51.029556 1166 ts_tablet_manager.cc:1918] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.029690 1166 log.cc:1198] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/41de7acfa7f941caa94bdbb888cf841e
I20250109 20:52:51.030030 1166 ts_tablet_manager.cc:1939] T 41de7acfa7f941caa94bdbb888cf841e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:51.030009 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2bdba403f4a24ce392324eeb74d3ed0e" } modify_external_catalogs: true
I20250109 20:52:51.030437 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1340076b6d8146429e90b485b3b58728"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.030848 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 41de7acfa7f941caa94bdbb888cf841e (table default.Table_65cb0c2ed23b450ca192c6f61becbc51 [id=2504d7b1b64f477a8b7763d5d0a9ec6e]) successfully deleted
W20250109 20:52:51.031441 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1340076b6d8146429e90b485b3b58728 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.032193 1176 tablet_bootstrap.cc:492] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.032248 1166 tablet_replica.cc:331] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:51.032375 1176 ts_tablet_manager.cc:1397] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.077s user 0.025s sys 0.035s
I20250109 20:52:51.033102 1176 raft_consensus.cc:357] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:51.033277 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c865faed94c3493ea638826bdcd4253d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c865faed94c3493ea638826bdcd4253d already in progress: creating tablet
I20250109 20:52:51.033303 1176 raft_consensus.cc:383] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.033429 1176 raft_consensus.cc:738] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.033681 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8d67b624984a0bb327d045c55476da with delete_type TABLET_DATA_DELETED (replaced by tablet 687db3e734af47339c2546970a71a539) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.033699 1176 consensus_queue.cc:260] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.034232 1176 ts_tablet_manager.cc:1428] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:51.034559 1176 tablet_bootstrap.cc:492] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.034746 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:51.036317 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d773b9e15aff4f4fa339a5c6c2e70bd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.037329 1176 tablet_bootstrap.cc:654] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:51.037353 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d773b9e15aff4f4fa339a5c6c2e70bd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.039304 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8d67b624984a0bb327d045c55476da with delete_type TABLET_DATA_DELETED (replaced by tablet 687db3e734af47339c2546970a71a539) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.040031 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ed658ec6109f45998eb6e916f76e8a79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.041209 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_66bc870f54384b4080562e0d880ce055" } new_table_name: "default.Table_01d2dd63e9a4480b961c408dc7d9a0a9" modify_external_catalogs: true
I20250109 20:52:51.042870 1158 tablet_bootstrap.cc:492] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.043026 1158 ts_tablet_manager.cc:1397] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250109 20:52:51.043388 448 catalog_manager.cc:6596] ReplaceTablet: tablet 3d8d67b624984a0bb327d045c55476da deleted and replaced by tablet 687db3e734af47339c2546970a71a539
I20250109 20:52:51.043697 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8d67b624984a0bb327d045c55476da with delete_type TABLET_DATA_DELETED (replaced by tablet 687db3e734af47339c2546970a71a539) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.044361 1158 raft_consensus.cc:357] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.044734 1158 raft_consensus.cc:383] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.045111 1176 tablet_bootstrap.cc:492] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.045106 1158 raft_consensus.cc:738] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.045277 1176 ts_tablet_manager.cc:1397] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20250109 20:52:51.045387 1158 consensus_queue.cc:260] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.046994 1158 ts_tablet_manager.cc:1428] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:51.047240 1176 raft_consensus.cc:357] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.047441 1176 raft_consensus.cc:383] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.047530 1176 raft_consensus.cc:738] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.047830 1158 tablet_bootstrap.cc:492] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:51.047742 1176 consensus_queue.cc:260] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.048183 1176 ts_tablet_manager.cc:1428] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:52:51.048444 1176 tablet_bootstrap.cc:492] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.048831 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d2ee28b720884b02a77177a6c829d85a" } modify_external_catalogs: true
I20250109 20:52:51.049015 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d2ee28b720884b02a77177a6c829d85a" } modify_external_catalogs: true
I20250109 20:52:51.050828 1176 tablet_bootstrap.cc:654] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.050832 1157 ts_tablet_manager.cc:1918] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.050922 1157 log.cc:1198] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/54d11b9148dc42bc91348e837af73972
I20250109 20:52:51.051268 1157 ts_tablet_manager.cc:1939] T 54d11b9148dc42bc91348e837af73972 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:51.052243 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 54d11b9148dc42bc91348e837af73972 (table default.Table_1e48be0e18b145708a9b6fa28ae21112 [id=194d1be4f3d946ec8c493ffcc5a23899]) successfully deleted
I20250109 20:52:51.055387 1176 tablet_bootstrap.cc:492] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.055526 1176 ts_tablet_manager.cc:1397] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250109 20:52:51.056170 1176 raft_consensus.cc:357] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.056365 1176 raft_consensus.cc:383] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.056427 1176 raft_consensus.cc:738] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.056648 1176 consensus_queue.cc:260] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.057077 1176 ts_tablet_manager.cc:1428] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:52:51.057339 1176 tablet_bootstrap.cc:492] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.059002 1176 tablet_bootstrap.cc:654] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.068589 1176 tablet_bootstrap.cc:492] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.068516 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a7a796c2d1d047baa518202a6b0091b7" } modify_external_catalogs: true
I20250109 20:52:51.068783 1176 ts_tablet_manager.cc:1397] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20250109 20:52:51.068751 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a7a796c2d1d047baa518202a6b0091b7" } modify_external_catalogs: true
I20250109 20:52:51.069479 1176 raft_consensus.cc:357] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.069669 1176 raft_consensus.cc:383] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.069665 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bb77198aaa4844c1a0dbd08621e876ca" } new_table_name: "default.Table_b153dbb4314c4300ac487674dda515eb" modify_external_catalogs: true
I20250109 20:52:51.069777 1176 raft_consensus.cc:738] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.070058 1176 consensus_queue.cc:260] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.070494 1176 ts_tablet_manager.cc:1428] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:51.070755 1176 tablet_bootstrap.cc:492] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.071442 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_01b9f979479d44219ae595df831ca7f8" } new_table_name: "default.Table_76d70836419947c78b06922a4fe21a2c" modify_external_catalogs: true
I20250109 20:52:51.072171 1176 tablet_bootstrap.cc:654] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:51.073386 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba consensus state has no leader
I20250109 20:52:51.073868 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2985f6de46ed417d87782a929b776d9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.075698 1176 tablet_bootstrap.cc:492] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.075986 1176 ts_tablet_manager.cc:1397] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250109 20:52:51.076759 1176 raft_consensus.cc:357] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.076968 1176 raft_consensus.cc:383] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.077029 1176 raft_consensus.cc:738] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.077214 1176 consensus_queue.cc:260] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.077582 1176 ts_tablet_manager.cc:1428] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:51.077848 1176 tablet_bootstrap.cc:492] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.079764 1176 tablet_bootstrap.cc:654] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.080142 1658 raft_consensus.cc:491] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.080266 1658 raft_consensus.cc:513] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.081080 1658 leader_election.cc:290] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:51.081115 1191 raft_consensus.cc:491] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.081247 1191 raft_consensus.cc:513] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.081681 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b06e8616df461d89c58d414fc0cfbd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:51.081835 1191 leader_election.cc:290] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:51.081990 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e16a1d9a5b374b8dbf725eb32ffadb9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e16a1d9a5b374b8dbf725eb32ffadb9e already in progress: creating tablet
W20250109 20:52:51.082147 989 leader_election.cc:343] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:51.082707 854 leader_election.cc:343] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: d08a3af516364157a07af7368a5f95f8
I20250109 20:52:51.084844 1158 tablet_bootstrap.cc:654] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.084962 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b06e8616df461d89c58d414fc0cfbd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:51.085500 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4f90c364ce844c9ea8eccb4ae23f8dce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f90c364ce844c9ea8eccb4ae23f8dce already in progress: creating tablet
W20250109 20:52:51.085502 990 leader_election.cc:343] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.085618 990 leader_election.cc:304] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:51.086115 1658 raft_consensus.cc:2743] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:51.087054 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c865faed94c3493ea638826bdcd4253d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c865faed94c3493ea638826bdcd4253d already in progress: creating tablet
W20250109 20:52:51.088325 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet fce062dc0ac144a19f8fa3eef4a099b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fce062dc0ac144a19f8fa3eef4a099b4 already in progress: creating tablet
I20250109 20:52:51.088407 1166 raft_consensus.cc:2238] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.088618 1166 raft_consensus.cc:2267] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.088824 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08a3af516364157a07af7368a5f95f8" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:51.089025 775 raft_consensus.cc:2463] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
W20250109 20:52:51.089437 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet dc235614171f4a61b75492ddeb3f0fe6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc235614171f4a61b75492ddeb3f0fe6 already in progress: creating tablet
I20250109 20:52:51.089685 1166 ts_tablet_manager.cc:1905] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.089785 854 leader_election.cc:304] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2; no voters: 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:52:51.090976 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4160bb4ba40347bda48f8fa88f8b6001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4160bb4ba40347bda48f8fa88f8b6001 already in progress: creating tablet
W20250109 20:52:51.091166 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1b50c91807734e86840e874886dc0749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b50c91807734e86840e874886dc0749 already in progress: creating tablet
W20250109 20:52:51.091297 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3d8d67b624984a0bb327d045c55476da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d8d67b624984a0bb327d045c55476da already in progress: creating tablet
I20250109 20:52:51.091537 1191 raft_consensus.cc:2798] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:51.091612 1191 raft_consensus.cc:491] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:51.091681 1191 raft_consensus.cc:3054] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:51.094650 1191 raft_consensus.cc:513] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.096304 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08a3af516364157a07af7368a5f95f8" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:51.096475 775 raft_consensus.cc:3054] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:51.098697 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_9f1c5bbd4e254d71b52e77fbaadb8b8b [id=ce415eef7536445b99cab5e0769d359b] alter complete (version 3)
I20250109 20:52:51.099081 775 raft_consensus.cc:2463] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:51.099581 854 leader_election.cc:304] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2; no voters:
I20250109 20:52:51.099679 1176 tablet_bootstrap.cc:492] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.099839 1176 ts_tablet_manager.cc:1397] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.002s
I20250109 20:52:51.100037 1657 raft_consensus.cc:2798] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:51.100621 1176 raft_consensus.cc:357] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.100855 1176 raft_consensus.cc:383] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.100935 1176 raft_consensus.cc:738] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.101186 1176 consensus_queue.cc:260] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.101908 1176 ts_tablet_manager.cc:1428] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:51.102161 1176 tablet_bootstrap.cc:492] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.104296 1176 tablet_bootstrap.cc:654] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:51.112429 854 leader_election.cc:343] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: d08a3af516364157a07af7368a5f95f8
I20250109 20:52:51.113076 1191 leader_election.cc:290] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:51.116058 1166 ts_tablet_manager.cc:1918] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.116228 1166 log.cc:1198] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/49c6f1833ff1453f90a6ec6435248a02
W20250109 20:52:51.120321 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2985f6de46ed417d87782a929b776d9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.121838 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed658ec6109f45998eb6e916f76e8a79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.129187 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b1891cbef8f4b0cae0c15d478db4860 consensus state has no leader
I20250109 20:52:51.131446 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e8ee4b86a04199865acd5f69cda384 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.131919 758 tablet_service.cc:1514] Processing DeleteTablet for tablet d08a3af516364157a07af7368a5f95f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:44 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.132194 1157 tablet_replica.cc:331] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:51.132436 1157 raft_consensus.cc:2238] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.132622 1157 raft_consensus.cc:2267] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.133353 1157 ts_tablet_manager.cc:1905] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.135031 1658 consensus_queue.cc:1035] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:51.137858 1657 raft_consensus.cc:695] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Running, Role: LEADER
I20250109 20:52:51.138264 1657 consensus_queue.cc:237] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.141669 1158 tablet_bootstrap.cc:492] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.143493 1158 ts_tablet_manager.cc:1397] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.006s
I20250109 20:52:51.147714 1157 ts_tablet_manager.cc:1918] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.147861 1157 log.cc:1198] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f0e8ee4b86a04199865acd5f69cda384
I20250109 20:52:51.148198 1157 ts_tablet_manager.cc:1939] T f0e8ee4b86a04199865acd5f69cda384 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:51.148106 1158 raft_consensus.cc:357] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.150161 1158 raft_consensus.cc:383] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.150314 1158 raft_consensus.cc:738] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.150661 1158 consensus_queue.cc:260] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.152907 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_067ec85f2e5e4ac1bd60cba79be3ac24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.153520 1158 ts_tablet_manager.cc:1428] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20250109 20:52:51.153769 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_067ec85f2e5e4ac1bd60cba79be3ac24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.155355 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5ad303eb4f07425d9b5aaa11eb265eea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.162362 1158 tablet_bootstrap.cc:492] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:51.162510 1157 tablet_replica.cc:331] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:51.162674 1157 raft_consensus.cc:2238] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.162863 1157 raft_consensus.cc:2267] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.168414 1158 tablet_bootstrap.cc:654] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.171972 1157 ts_tablet_manager.cc:1905] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.180959 1677 raft_consensus.cc:491] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.181167 1677 raft_consensus.cc:513] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.182107 1677 leader_election.cc:290] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:51.183725 712 leader_election.cc:343] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Not found: Tablet not found: 46a9910f88574eac90db9e00138c8dda
W20250109 20:52:51.184423 713 leader_election.cc:343] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 46a9910f88574eac90db9e00138c8dda
I20250109 20:52:51.184587 713 leader_election.cc:304] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:51.185286 1677 raft_consensus.cc:2743] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.193480 1658 consensus_queue.cc:1035] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:51.202298 1681 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:51.203927 1683 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:51.161940 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f0e8ee4b86a04199865acd5f69cda384 (table default.Table_b683408175de45c0bf7d7f7e14e15447 [id=2a713bcff6ae4df0a2a9ad75a6986394]) successfully deleted
W20250109 20:52:51.207422 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 45505.4us
I20250109 20:52:51.195659 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5377c27f242f4aaf8ef9d0281640a856"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.209533 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5377c27f242f4aaf8ef9d0281640a856 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.212299 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ad303eb4f07425d9b5aaa11eb265eea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.213999 1158 tablet_bootstrap.cc:492] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.214210 1158 ts_tablet_manager.cc:1397] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.004s
I20250109 20:52:51.215142 1158 raft_consensus.cc:357] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.216344 1158 raft_consensus.cc:383] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.216580 1158 raft_consensus.cc:738] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.217484 1158 consensus_queue.cc:260] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.218804 1158 ts_tablet_manager.cc:1428] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:51.219826 1158 tablet_bootstrap.cc:492] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:51.220654 1704 raft_consensus.cc:491] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.221650 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:51.223289 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_141b5c9b04d6457f8ebe492f7a7fb482"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.225087 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_141b5c9b04d6457f8ebe492f7a7fb482 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.229795 1704 raft_consensus.cc:513] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.230070 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_24b4edbc6e754ee1aeb45736c5b4aa8e" } new_table_name: "default.Table_54acd558d6f04df09ecc0669cabfe878" modify_external_catalogs: true
I20250109 20:52:51.232466 1704 leader_election.cc:290] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:51.233359 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab7ca098694464faec002cec05558ac" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:51.233479 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab7ca098694464faec002cec05558ac" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:51.234001 1703 tablet.cc:1721] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:51.236593 1157 ts_tablet_manager.cc:1918] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.236734 1157 log.cc:1198] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/d08a3af516364157a07af7368a5f95f8
I20250109 20:52:51.238859 1158 tablet_bootstrap.cc:654] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.240079 1157 ts_tablet_manager.cc:1939] T d08a3af516364157a07af7368a5f95f8 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:52:51.240170 989 leader_election.cc:343] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.253199 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet d08a3af516364157a07af7368a5f95f8 (table default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7 [id=ada5a5d915bf40e884cbd0e86a8e5c31]) successfully deleted
I20250109 20:52:51.255467 1158 tablet_bootstrap.cc:492] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.255657 1158 ts_tablet_manager.cc:1397] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20250109 20:52:51.256423 1158 raft_consensus.cc:357] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.256649 1158 raft_consensus.cc:383] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.256765 1158 raft_consensus.cc:738] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:51.256791 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 2ed7abf23ce04a268a460d8e3e97d56b (table default.Table_5e6c78d0dceb4466bb81f225c45bee0d [id=b296e8173f88419a8a14ac395611b4bc], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2ed7abf23ce04a268a460d8e3e97d56b consensus state has no leader
I20250109 20:52:51.257040 1158 consensus_queue.cc:260] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.257516 1158 ts_tablet_manager.cc:1428] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:51.258008 1158 tablet_bootstrap.cc:492] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:51.260792 1158 tablet_bootstrap.cc:654] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.263764 1160 ts_tablet_manager.cc:1918] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:51.264262 990 leader_election.cc:343] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.264609 990 leader_election.cc:304] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:51.264937 1704 raft_consensus.cc:2743] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.277909 1166 ts_tablet_manager.cc:1939] T 49c6f1833ff1453f90a6ec6435248a02 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:51.279014 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 49c6f1833ff1453f90a6ec6435248a02 (table default.Table_d59d31f7bc514ff38a99d27cbe8170f9 [id=5514c15f7b8c406288bec6394be1d99b]) successfully deleted
W20250109 20:52:51.279484 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet dc235614171f4a61b75492ddeb3f0fe6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc235614171f4a61b75492ddeb3f0fe6 already in progress: creating tablet
W20250109 20:52:51.279659 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1b50c91807734e86840e874886dc0749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b50c91807734e86840e874886dc0749 already in progress: creating tablet
W20250109 20:52:51.279824 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4160bb4ba40347bda48f8fa88f8b6001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4160bb4ba40347bda48f8fa88f8b6001 already in progress: creating tablet
W20250109 20:52:51.280237 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3d8d67b624984a0bb327d045c55476da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d8d67b624984a0bb327d045c55476da already in progress: creating tablet
I20250109 20:52:51.287693 1677 raft_consensus.cc:491] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.287884 1677 raft_consensus.cc:513] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.288659 1677 leader_election.cc:290] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:51.289746 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4f256d59a2459e95be0f950a5a86eb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:51.270150 1703 tablet.cc:1721] T da2d6dda69da4a59b055106c1041ec2b P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:51.292610 1160 log.cc:1198] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/08ddedf78c0442daa9b1d05334f12b4d
I20250109 20:52:51.293097 1160 ts_tablet_manager.cc:1939] T 08ddedf78c0442daa9b1d05334f12b4d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:51.297864 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:51.291860 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4f256d59a2459e95be0f950a5a86eb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:51.301725 919 raft_consensus.cc:2388] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09833ea46b7141d58c3899dace69e49f in current term 1: Already voted for candidate 5fcc647ed479491b88e5ffacf2927a1a in this term.
I20250109 20:52:51.302310 712 leader_election.cc:304] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:51.302930 1678 raft_consensus.cc:3054] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:51.306373 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dcd777ce1ff24c3db7dafe776cf3b871" } new_table_name: "default.Table_10569a5c687543ae82b16452e4cb316f" modify_external_catalogs: true
I20250109 20:52:51.329983 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c213e6faecce454c879d19b0bd0e787a" } modify_external_catalogs: true
I20250109 20:52:51.330317 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c213e6faecce454c879d19b0bd0e787a" } modify_external_catalogs: true
I20250109 20:52:51.331969 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_eb2e935fd7fb43c0ba59c503c575d803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.332738 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb2e935fd7fb43c0ba59c503c575d803 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.334317 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbef1d27c5394fc1b6354c5d6f3a8f1f has no consensus state
I20250109 20:52:51.334785 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_937a1971308f4f759b74d375c3808729"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.335520 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_937a1971308f4f759b74d375c3808729 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.351718 1678 raft_consensus.cc:2743] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.355475 1165 tablet_bootstrap.cc:492] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:51.355657 1165 ts_tablet_manager.cc:1397] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.447s user 0.007s sys 0.005s
I20250109 20:52:51.357002 1165 raft_consensus.cc:357] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.357239 1165 raft_consensus.cc:383] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.357354 1165 raft_consensus.cc:738] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.357601 1165 consensus_queue.cc:260] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.358477 1165 ts_tablet_manager.cc:1428] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:51.358537 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 08ddedf78c0442daa9b1d05334f12b4d (table default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b [id=f2abcc91ea094f11a4d434ae57e6f404]) successfully deleted
I20250109 20:52:51.359239 1165 tablet_bootstrap.cc:492] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:51.360826 1165 tablet_bootstrap.cc:654] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:51.361104 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4f90c364ce844c9ea8eccb4ae23f8dce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f90c364ce844c9ea8eccb4ae23f8dce already in progress: creating tablet
W20250109 20:52:51.361806 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c865faed94c3493ea638826bdcd4253d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c865faed94c3493ea638826bdcd4253d already in progress: creating tablet
W20250109 20:52:51.362514 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet fce062dc0ac144a19f8fa3eef4a099b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fce062dc0ac144a19f8fa3eef4a099b4 already in progress: creating tablet
W20250109 20:52:51.363270 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e16a1d9a5b374b8dbf725eb32ffadb9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e16a1d9a5b374b8dbf725eb32ffadb9e already in progress: creating tablet
I20250109 20:52:51.363688 1160 tablet_replica.cc:331] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:51.363971 1160 raft_consensus.cc:2238] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:51.364295 1160 pending_rounds.cc:62] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Trying to abort 1 pending ops.
I20250109 20:52:51.364434 1160 pending_rounds.cc:69] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523657785298944 op_type: NO_OP noop_request { }
I20250109 20:52:51.364634 1160 raft_consensus.cc:2883] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:51.364841 1160 raft_consensus.cc:2267] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.368589 1160 ts_tablet_manager.cc:1905] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.383893 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
W20250109 20:52:51.384192 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d1630e4438e4337ba026b41f630a480 consensus state has no leader
I20250109 20:52:51.385192 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_98192f0dc90240989c8b4c8c972d7751"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.385370 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bfad8a48292647e68716ec5438e5a6a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.385813 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98192f0dc90240989c8b4c8c972d7751 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.385813 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfad8a48292647e68716ec5438e5a6a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.390017 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_01fe90e030ba4f5cac35ad1e4797122d" } new_table_name: "default.Table_d5fcf8d73704433193c50e16d0fe4184" modify_external_catalogs: true
I20250109 20:52:51.389009 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_76d70836419947c78b06922a4fe21a2c [id=53718e55e4344456851f408909cc8170] alter complete (version 4)
I20250109 20:52:51.392102 758 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd6ce78438841daa663b6344f062019 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.392408 1157 tablet_replica.cc:331] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:51.392668 1157 raft_consensus.cc:2238] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.392910 1157 raft_consensus.cc:2267] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.393591 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a9910f88574eac90db9e00138c8dda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.393616 1157 ts_tablet_manager.cc:1905] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.396191 897 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd6ce78438841daa663b6344f062019 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.401042 1157 ts_tablet_manager.cc:1918] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.401216 1157 log.cc:1198] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ffd6ce78438841daa663b6344f062019
I20250109 20:52:51.401643 1157 ts_tablet_manager.cc:1939] T ffd6ce78438841daa663b6344f062019 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:51.402806 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ffd6ce78438841daa663b6344f062019 (table default.Table_d8369c071899441a9b05f8910c9743c2 [id=b939987c4a0f426b846403a8579622c7]) successfully deleted
I20250109 20:52:51.403259 1157 tablet_replica.cc:331] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:51.403493 1157 raft_consensus.cc:2238] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.403752 1157 raft_consensus.cc:2267] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.404536 1157 ts_tablet_manager.cc:1905] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.417276 1157 ts_tablet_manager.cc:1918] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.419512 1157 log.cc:1198] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/46a9910f88574eac90db9e00138c8dda
I20250109 20:52:51.420089 1157 ts_tablet_manager.cc:1939] T 46a9910f88574eac90db9e00138c8dda P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:51.421672 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 46a9910f88574eac90db9e00138c8dda (table default.Table_9d14aaae5e2d40a3842ad98fabf33015 [id=1e8fda0dc6034fdf9aa9b52712005046]) successfully deleted
I20250109 20:52:51.442584 1451 raft_consensus.cc:491] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.442874 1451 raft_consensus.cc:513] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.444206 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e90cd406e394166b3e8dedebf7fd0fc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:51.444885 990 leader_election.cc:343] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.445976 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e90cd406e394166b3e8dedebf7fd0fc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:51.446525 989 leader_election.cc:343] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.446652 989 leader_election.cc:304] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:51.446913 1451 leader_election.cc:290] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:51.447227 1451 raft_consensus.cc:2743] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.466416 1160 ts_tablet_manager.cc:1918] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:51.466652 1160 log.cc:1198] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/d08a3af516364157a07af7368a5f95f8
I20250109 20:52:51.468348 1160 ts_tablet_manager.cc:1939] T d08a3af516364157a07af7368a5f95f8 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:51.469025 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet d08a3af516364157a07af7368a5f95f8 (table default.Table_e1ace4a12f5d42f3a73d605f98ba7ac7 [id=ada5a5d915bf40e884cbd0e86a8e5c31]) successfully deleted
W20250109 20:52:51.469837 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1b50c91807734e86840e874886dc0749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b50c91807734e86840e874886dc0749 already in progress: creating tablet
I20250109 20:52:51.470561 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a137a6b3e0614575b0a4f169e928ce95" } new_table_name: "default.Table_b856f34058994714afb4ce41d44f5384" modify_external_catalogs: true
W20250109 20:52:51.470760 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet dc235614171f4a61b75492ddeb3f0fe6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc235614171f4a61b75492ddeb3f0fe6 already in progress: creating tablet
I20250109 20:52:51.471298 897 tablet_service.cc:1467] Processing CreateTablet for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f (DEFAULT_TABLE table=default.Table_54acd558d6f04df09ecc0669cabfe878 [id=81f954b2a3874b34b98f34d39a126140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.471635 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.472826 744 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6ec8f3d4f4d37a11f15173449155c (DEFAULT_TABLE table=default.Table_2bdba403f4a24ce392324eeb74d3ed0e [id=ab5d37b49fc549f8a7829a858ea0b3bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.473191 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6ec8f3d4f4d37a11f15173449155c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.473320 758 tablet_service.cc:1467] Processing CreateTablet for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f (DEFAULT_TABLE table=default.Table_54acd558d6f04df09ecc0669cabfe878 [id=81f954b2a3874b34b98f34d39a126140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.473589 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.475641 749 tablet_service.cc:1467] Processing CreateTablet for tablet 40546e1b5e854a79a7f706a43b6c6062 (DEFAULT_TABLE table=default.Table_6c79e0eceb7b4c6a8c751387a1a7118a [id=3b1b495828aa4de2b1362a99304baedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.475927 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40546e1b5e854a79a7f706a43b6c6062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.475891 1023 tablet_service.cc:1467] Processing CreateTablet for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f (DEFAULT_TABLE table=default.Table_54acd558d6f04df09ecc0669cabfe878 [id=81f954b2a3874b34b98f34d39a126140]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.476256 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.476506 742 tablet_service.cc:1467] Processing CreateTablet for tablet 189cd8079a9d4ca6b1703f4361d5ec5d (DEFAULT_TABLE table=default.Table_7de47568081c4d11a33a67edd6898f8f [id=f4be51ab8d8b4a1a9238e74cb1d3e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.476835 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189cd8079a9d4ca6b1703f4361d5ec5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.478268 883 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6ec8f3d4f4d37a11f15173449155c (DEFAULT_TABLE table=default.Table_2bdba403f4a24ce392324eeb74d3ed0e [id=ab5d37b49fc549f8a7829a858ea0b3bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.478569 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6ec8f3d4f4d37a11f15173449155c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.479969 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6ec8f3d4f4d37a11f15173449155c (DEFAULT_TABLE table=default.Table_2bdba403f4a24ce392324eeb74d3ed0e [id=ab5d37b49fc549f8a7829a858ea0b3bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.480249 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6ec8f3d4f4d37a11f15173449155c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.480933 893 tablet_service.cc:1467] Processing CreateTablet for tablet 40546e1b5e854a79a7f706a43b6c6062 (DEFAULT_TABLE table=default.Table_6c79e0eceb7b4c6a8c751387a1a7118a [id=3b1b495828aa4de2b1362a99304baedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.481218 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40546e1b5e854a79a7f706a43b6c6062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.484825 896 tablet_service.cc:1467] Processing CreateTablet for tablet 189cd8079a9d4ca6b1703f4361d5ec5d (DEFAULT_TABLE table=default.Table_7de47568081c4d11a33a67edd6898f8f [id=f4be51ab8d8b4a1a9238e74cb1d3e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.485002 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 40546e1b5e854a79a7f706a43b6c6062 (DEFAULT_TABLE table=default.Table_6c79e0eceb7b4c6a8c751387a1a7118a [id=3b1b495828aa4de2b1362a99304baedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.484968 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_98b8f1400db644be881e04403ed16d10" } new_table_name: "default.Table_814844402c5e4defbb723b986171a2a2" modify_external_catalogs: true
I20250109 20:52:51.485338 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40546e1b5e854a79a7f706a43b6c6062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.485528 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189cd8079a9d4ca6b1703f4361d5ec5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.486800 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:51.488267 447 catalog_manager.cc:5526] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
I20250109 20:52:51.488834 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:51.490274 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_46b0126178104280ad6c27f0c3001a00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.490837 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46b0126178104280ad6c27f0c3001a00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.490891 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 189cd8079a9d4ca6b1703f4361d5ec5d (DEFAULT_TABLE table=default.Table_7de47568081c4d11a33a67edd6898f8f [id=f4be51ab8d8b4a1a9238e74cb1d3e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.491223 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189cd8079a9d4ca6b1703f4361d5ec5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.491777 890 tablet_service.cc:1467] Processing CreateTablet for tablet a64e619e3e7e49469adca01d95d75d02 (DEFAULT_TABLE table=default.Table_9a6081630d9c4441916bca2f2d06ce63 [id=424b6f561c664895b836e7d3944c6b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.492233 898 tablet_service.cc:1467] Processing CreateTablet for tablet c83b61e879fc4979a85c872f2e6ab88c (DEFAULT_TABLE table=default.Table_5cf48a44b13a4059b1e8974875e736e2 [id=7080faa7b3844702859c8a48be032b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.493059 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c83b61e879fc4979a85c872f2e6ab88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.494442 880 tablet_service.cc:1467] Processing CreateTablet for tablet edc803f2936b4538a5e70ef34e5cc5d4 (DEFAULT_TABLE table=default.Table_f456c719d70549da87d54647fd2d02aa [id=d969b9f61388495f90e0351621b658e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.494756 888 tablet_service.cc:1467] Processing CreateTablet for tablet 80faa29a3b314796b68a4073665d2898 (DEFAULT_TABLE table=default.Table_283db7be437c43d8845b9f6d4d906bc7 [id=981a80b153d644bcab94aaaaa9aa6e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.495213 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc803f2936b4538a5e70ef34e5cc5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.497081 1035 tablet_service.cc:1467] Processing CreateTablet for tablet a64e619e3e7e49469adca01d95d75d02 (DEFAULT_TABLE table=default.Table_9a6081630d9c4441916bca2f2d06ce63 [id=424b6f561c664895b836e7d3944c6b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.497495 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64e619e3e7e49469adca01d95d75d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.501127 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 80faa29a3b314796b68a4073665d2898 (DEFAULT_TABLE table=default.Table_283db7be437c43d8845b9f6d4d906bc7 [id=981a80b153d644bcab94aaaaa9aa6e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.501538 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80faa29a3b314796b68a4073665d2898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.504808 885 tablet_service.cc:1467] Processing CreateTablet for tablet f1534c7240fc49f684af61aef783ee24 (DEFAULT_TABLE table=default.Table_6c195088c8764cc2a94609872588c4a4 [id=97a35961c6c24ff6bee932ed763714b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.505137 1029 tablet_service.cc:1467] Processing CreateTablet for tablet c83b61e879fc4979a85c872f2e6ab88c (DEFAULT_TABLE table=default.Table_5cf48a44b13a4059b1e8974875e736e2 [id=7080faa7b3844702859c8a48be032b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.505565 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c83b61e879fc4979a85c872f2e6ab88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.506788 895 tablet_service.cc:1467] Processing CreateTablet for tablet feb9b0f722eb44b0acd5cb3bbb32c527 (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.507018 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80faa29a3b314796b68a4073665d2898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.507388 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb9b0f722eb44b0acd5cb3bbb32c527. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.508425 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64e619e3e7e49469adca01d95d75d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.509063 1016 tablet_service.cc:1467] Processing CreateTablet for tablet edc803f2936b4538a5e70ef34e5cc5d4 (DEFAULT_TABLE table=default.Table_f456c719d70549da87d54647fd2d02aa [id=d969b9f61388495f90e0351621b658e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.509441 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc803f2936b4538a5e70ef34e5cc5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.510828 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1534c7240fc49f684af61aef783ee24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.514003 742 tablet_service.cc:1467] Processing CreateTablet for tablet a64e619e3e7e49469adca01d95d75d02 (DEFAULT_TABLE table=default.Table_9a6081630d9c4441916bca2f2d06ce63 [id=424b6f561c664895b836e7d3944c6b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.514449 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64e619e3e7e49469adca01d95d75d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.514401 739 tablet_service.cc:1467] Processing CreateTablet for tablet feb9b0f722eb44b0acd5cb3bbb32c527 (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.514732 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb9b0f722eb44b0acd5cb3bbb32c527. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.514948 744 tablet_service.cc:1467] Processing CreateTablet for tablet 80faa29a3b314796b68a4073665d2898 (DEFAULT_TABLE table=default.Table_283db7be437c43d8845b9f6d4d906bc7 [id=981a80b153d644bcab94aaaaa9aa6e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.515262 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80faa29a3b314796b68a4073665d2898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.518255 1032 tablet_service.cc:1467] Processing CreateTablet for tablet f1534c7240fc49f684af61aef783ee24 (DEFAULT_TABLE table=default.Table_6c195088c8764cc2a94609872588c4a4 [id=97a35961c6c24ff6bee932ed763714b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.518538 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1534c7240fc49f684af61aef783ee24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.518553 758 tablet_service.cc:1467] Processing CreateTablet for tablet c83b61e879fc4979a85c872f2e6ab88c (DEFAULT_TABLE table=default.Table_5cf48a44b13a4059b1e8974875e736e2 [id=7080faa7b3844702859c8a48be032b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.518817 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c83b61e879fc4979a85c872f2e6ab88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.518218 1020 tablet_service.cc:1467] Processing CreateTablet for tablet feb9b0f722eb44b0acd5cb3bbb32c527 (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.520963 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb9b0f722eb44b0acd5cb3bbb32c527. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.522867 749 tablet_service.cc:1467] Processing CreateTablet for tablet edc803f2936b4538a5e70ef34e5cc5d4 (DEFAULT_TABLE table=default.Table_f456c719d70549da87d54647fd2d02aa [id=d969b9f61388495f90e0351621b658e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.523133 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc803f2936b4538a5e70ef34e5cc5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.528394 756 tablet_service.cc:1467] Processing CreateTablet for tablet f1534c7240fc49f684af61aef783ee24 (DEFAULT_TABLE table=default.Table_6c195088c8764cc2a94609872588c4a4 [id=97a35961c6c24ff6bee932ed763714b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.528728 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1534c7240fc49f684af61aef783ee24. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:51.548640 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59ebeffe516c4c239ae2ee9040276e53 consensus state has no leader
I20250109 20:52:51.549465 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0477b4a373d5435fac417a7c26c5103c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.550000 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0477b4a373d5435fac417a7c26c5103c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.550354 1702 raft_consensus.cc:491] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.552109 1702 raft_consensus.cc:513] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.556144 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acca65ecee94f8cbe3f2b89e744aa6c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:51.556778 990 leader_election.cc:343] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.557742 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acca65ecee94f8cbe3f2b89e744aa6c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:51.558351 989 leader_election.cc:343] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.558502 989 leader_election.cc:304] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:51.559484 1704 raft_consensus.cc:491] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.559635 1704 raft_consensus.cc:513] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.560338 1704 leader_election.cc:290] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:51.560518 1704 raft_consensus.cc:2743] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.561194 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a89a0b47d10414d9b6a9123af1f4ab3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:51.561578 1158 tablet_bootstrap.cc:492] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.561714 1158 ts_tablet_manager.cc:1397] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.304s user 0.023s sys 0.052s
W20250109 20:52:51.561755 990 leader_election.cc:343] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.562428 1158 raft_consensus.cc:357] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.562614 1158 raft_consensus.cc:383] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.562681 1158 raft_consensus.cc:738] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.562695 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a89a0b47d10414d9b6a9123af1f4ab3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:51.562880 1158 consensus_queue.cc:260] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.563330 1158 ts_tablet_manager.cc:1428] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:51.563619 1158 tablet_bootstrap.cc:492] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:51.564320 1704 consensus_queue.cc:1035] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:51.565692 1158 tablet_bootstrap.cc:654] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:51.566344 989 leader_election.cc:343] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.566496 989 leader_election.cc:304] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:51.567030 1704 raft_consensus.cc:2743] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.569046 1702 leader_election.cc:290] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:51.578230 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffd3a1b1d3e44feda6dbaa8556e377fc consensus state has no leader
I20250109 20:52:51.579787 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 7df7405e93aa4718a62933ae56af9f42 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:51.582027 919 raft_consensus.cc:3054] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:51.582962 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.585472 1157 tablet_replica.cc:331] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:51.585717 1157 raft_consensus.cc:2238] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.585757 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 22f6ec8f3d4f4d37a11f15173449155c
I20250109 20:52:51.585889 1157 raft_consensus.cc:2267] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.586197 1704 raft_consensus.cc:491] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:52:51.586341 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4160bb4ba40347bda48f8fa88f8b6001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4160bb4ba40347bda48f8fa88f8b6001 already in progress: creating tablet
I20250109 20:52:51.586337 1704 raft_consensus.cc:513] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.586828 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6ec8f3d4f4d37a11f15173449155c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.586989 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 47726b58d6714cd1b46848a16a34ff37
I20250109 20:52:51.587050 1704 leader_election.cc:290] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:51.587536 1157 ts_tablet_manager.cc:1905] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.587725 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6fb563bd6802473fafedd08bb10d15c1
I20250109 20:52:51.587787 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540531463c184b9abf226d7487939e60" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:51.588397 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9170b3a6b9bb446b8a474ae4df249409
W20250109 20:52:51.588625 990 leader_election.cc:343] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.590440 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "540531463c184b9abf226d7487939e60" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:51.590427 1704 consensus_queue.cc:1035] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:52:51.591432 989 leader_election.cc:343] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.591583 989 leader_election.cc:304] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:51.593518 1704 raft_consensus.cc:2743] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.593843 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9170b3a6b9bb446b8a474ae4df249409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.594193 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.594374 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 47726b58d6714cd1b46848a16a34ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.594614 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.594794 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9170b3a6b9bb446b8a474ae4df249409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.595037 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6ec8f3d4f4d37a11f15173449155c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:51.595727 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9170b3a6b9bb446b8a474ae4df249409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9170b3a6b9bb446b8a474ae4df249409 already in progress: creating tablet
W20250109 20:52:51.595937 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6fb563bd6802473fafedd08bb10d15c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fb563bd6802473fafedd08bb10d15c1 already in progress: creating tablet
I20250109 20:52:51.597759 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9170b3a6b9bb446b8a474ae4df249409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.598067 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.598318 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 47726b58d6714cd1b46848a16a34ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.598585 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 47726b58d6714cd1b46848a16a34ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:51.599362 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 47726b58d6714cd1b46848a16a34ff37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47726b58d6714cd1b46848a16a34ff37 already in progress: creating tablet
I20250109 20:52:51.599783 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6ec8f3d4f4d37a11f15173449155c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.600319 1157 ts_tablet_manager.cc:1918] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.600448 1157 log.cc:1198] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/7d981c7f13c3409f994bfdf0275e0116
W20250109 20:52:51.600603 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 22f6ec8f3d4f4d37a11f15173449155c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22f6ec8f3d4f4d37a11f15173449155c already in progress: creating tablet
I20250109 20:52:51.600780 1157 ts_tablet_manager.cc:1939] T 7d981c7f13c3409f994bfdf0275e0116 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:51.601898 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 7d981c7f13c3409f994bfdf0275e0116 (table default.Table_2acc983c759e437cbce5b1a3a5e88cfe [id=b64c15100a8344a6bc35a896aec8270c]) successfully deleted
W20250109 20:52:51.605319 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 47726b58d6714cd1b46848a16a34ff37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47726b58d6714cd1b46848a16a34ff37 already in progress: creating tablet
W20250109 20:52:51.606120 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6fb563bd6802473fafedd08bb10d15c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fb563bd6802473fafedd08bb10d15c1 already in progress: creating tablet
W20250109 20:52:51.606794 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9170b3a6b9bb446b8a474ae4df249409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9170b3a6b9bb446b8a474ae4df249409 already in progress: creating tablet
W20250109 20:52:51.607443 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 22f6ec8f3d4f4d37a11f15173449155c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22f6ec8f3d4f4d37a11f15173449155c already in progress: creating tablet
W20250109 20:52:51.608284 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3d8d67b624984a0bb327d045c55476da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d8d67b624984a0bb327d045c55476da already in progress: creating tablet
I20250109 20:52:51.608530 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f2c83e709a8d4e078e67af58ec51beac
I20250109 20:52:51.609253 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5ad303eb4f07425d9b5aaa11eb265eea" } new_table_name: "default.Table_7193fd56447746c98aef50288388883c" modify_external_catalogs: true
I20250109 20:52:51.609325 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c83e709a8d4e078e67af58ec51beac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.609668 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e4be594b7e46489d83d7c7f8bbbed681"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.610109 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4be594b7e46489d83d7c7f8bbbed681 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.610590 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75cdf5b737504d4e8f598cd2b265192e consensus state has no leader
I20250109 20:52:51.610842 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 73e82535780c47ca8b9cd94b013b4b42 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:51.611318 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a0f0901b5b4e4116ac67c1bcc180251a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.611445 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c83e709a8d4e078e67af58ec51beac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:51.611766 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0f0901b5b4e4116ac67c1bcc180251a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.612002 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f2c83e709a8d4e078e67af58ec51beac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2c83e709a8d4e078e67af58ec51beac already in progress: creating tablet
I20250109 20:52:51.612382 758 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c83e709a8d4e078e67af58ec51beac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:51.612990 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f2c83e709a8d4e078e67af58ec51beac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2c83e709a8d4e078e67af58ec51beac already in progress: creating tablet
I20250109 20:52:51.613477 1160 tablet_replica.cc:331] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:51.613713 1160 raft_consensus.cc:2238] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.613888 1160 raft_consensus.cc:2267] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.615110 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_68df6c0c372443ba892b057eacff3ffa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.615514 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68df6c0c372443ba892b057eacff3ffa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.616667 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e59899b8fce64608ac583be8037520ca" } new_table_name: "default.Table_113e0f4bda2b441986056ee8e3616d94" modify_external_catalogs: true
I20250109 20:52:51.621011 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6bcb56e269714b399e7d1fbc2e2c290e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.621424 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bcb56e269714b399e7d1fbc2e2c290e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.621691 1160 ts_tablet_manager.cc:1905] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.622815 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f54840d0459f412da09f1eb1bd75ae74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.623322 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f54840d0459f412da09f1eb1bd75ae74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.624269 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_47985b842e6c4ab6adaf55ef480a90d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.624460 1165 tablet_bootstrap.cc:492] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:51.624576 1165 ts_tablet_manager.cc:1397] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.265s user 0.037s sys 0.071s
W20250109 20:52:51.624889 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47985b842e6c4ab6adaf55ef480a90d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.625402 1165 raft_consensus.cc:357] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.625591 1165 raft_consensus.cc:383] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.625655 1165 raft_consensus.cc:738] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.626509 1165 consensus_queue.cc:260] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.626674 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:51.626997 1165 ts_tablet_manager.cc:1428] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:51.627269 1165 tablet_bootstrap.cc:492] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:51.630631 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:51.632746 1165 tablet_bootstrap.cc:654] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.633320 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet dae574bf93d649099252e9e7b09b748c with delete_type TABLET_DATA_DELETED (replaced by tablet dc235614171f4a61b75492ddeb3f0fe6) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.633549 1166 tablet_replica.cc:331] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:51.633725 1166 raft_consensus.cc:2238] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.633882 1166 raft_consensus.cc:2267] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.634414 1166 ts_tablet_manager.cc:1905] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.640642 1166 ts_tablet_manager.cc:1918] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.642367 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:51.645185 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:51.645881 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 5b2fcddba2ac4d82b2d25bfc64a5967e from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:51.647449 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.647900 1158 tablet_bootstrap.cc:492] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.648062 1158 ts_tablet_manager.cc:1397] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.000s
W20250109 20:52:51.648280 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7d981c7f13c3409f994bfdf0275e0116 because the tablet was not found. No further retry: Not found: Tablet not found: 7d981c7f13c3409f994bfdf0275e0116
I20250109 20:52:51.648932 1158 raft_consensus.cc:357] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.650173 1158 raft_consensus.cc:383] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.650261 1158 raft_consensus.cc:738] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.654074 1158 consensus_queue.cc:260] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.655822 1158 ts_tablet_manager.cc:1428] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250109 20:52:51.656101 1158 tablet_bootstrap.cc:492] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:51.662391 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e9041a2e91f4753aa561965cc0e6280 has no consensus state
I20250109 20:52:51.662727 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:51.662961 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 8890a6a93e3d48dc96955f1b51e78603 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:51.664245 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d981c7f13c3409f994bfdf0275e0116 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:51.664350 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8eba3dcab764a98857604b63bf9dce1 consensus state has no leader
I20250109 20:52:51.665649 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 with delete_type TABLET_DATA_DELETED (replaced by tablet e64207928ef141118e7fc8fe7bb2379b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.665918 1157 tablet_replica.cc:331] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:51.666072 1157 raft_consensus.cc:2238] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.666229 1157 raft_consensus.cc:2267] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.667557 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7df7405e93aa4718a62933ae56af9f42
I20250109 20:52:51.668488 453 catalog_manager.cc:6596] ReplaceTablet: tablet 7df7405e93aa4718a62933ae56af9f42 deleted and replaced by tablet 8f77f35085ff43a5b7fb6cce7e7a3409
I20250109 20:52:51.671083 1711 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:52:51.671232 1711 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:52:51.671957 1711 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
I20250109 20:52:51.672827 1158 tablet_bootstrap.cc:654] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.675290 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df7405e93aa4718a62933ae56af9f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f77f35085ff43a5b7fb6cce7e7a3409) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.676877 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 651 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:52:51.677348 1157 ts_tablet_manager.cc:1905] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.677563 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df7405e93aa4718a62933ae56af9f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f77f35085ff43a5b7fb6cce7e7a3409) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.680997 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 651 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:52:51.681905 1400 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
I20250109 20:52:51.682335 1711 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:52:51.683458 1157 ts_tablet_manager.cc:1918] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.683615 1157 log.cc:1198] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2b9feb9de9dc4d22bafc0f7707943113
I20250109 20:52:51.684036 1157 ts_tablet_manager.cc:1939] T 2b9feb9de9dc4d22bafc0f7707943113 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:51.685046 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2b9feb9de9dc4d22bafc0f7707943113 (table default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]) successfully deleted
W20250109 20:52:51.685700 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7df7405e93aa4718a62933ae56af9f42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7df7405e93aa4718a62933ae56af9f42 already in progress: creating tablet
I20250109 20:52:51.686897 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7d9492ed2e5b44ada92f6fa5faa03ab9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.687422 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d9492ed2e5b44ada92f6fa5faa03ab9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.687561 1166 log.cc:1198] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/dae574bf93d649099252e9e7b09b748c
I20250109 20:52:51.688072 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:51.689953 1176 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.084s sys 0.154s
W20250109 20:52:51.690178 1176 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.584s user 0.084s sys 0.155s
I20250109 20:52:51.690438 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 540531463c184b9abf226d7487939e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.691102 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df7405e93aa4718a62933ae56af9f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f77f35085ff43a5b7fb6cce7e7a3409) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.691208 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.690560 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 3000b819a4cc4960909b98febd400631 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c65489a602141239c99e8e8dae22ab4) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.691435 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c4ae127254154c808ee79dd8e20f184a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.691869 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4ae127254154c808ee79dd8e20f184a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.692253 1166 ts_tablet_manager.cc:1939] T dae574bf93d649099252e9e7b09b748c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:51.693266 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet dae574bf93d649099252e9e7b09b748c (table default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]) successfully deleted
I20250109 20:52:51.693476 1166 tablet_replica.cc:331] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:51.693655 1166 raft_consensus.cc:2238] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.694042 1166 raft_consensus.cc:2267] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.694536 1166 ts_tablet_manager.cc:1905] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.697088 1165 tablet_bootstrap.cc:492] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:51.697178 1176 tablet_bootstrap.cc:492] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.697250 1165 ts_tablet_manager.cc:1397] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.002s
I20250109 20:52:51.697316 1176 ts_tablet_manager.cc:1397] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.595s user 0.085s sys 0.156s
I20250109 20:52:51.697994 1176 raft_consensus.cc:357] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.698172 1176 raft_consensus.cc:383] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.698102 1165 raft_consensus.cc:357] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.698233 1176 raft_consensus.cc:738] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.698272 1165 raft_consensus.cc:383] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.698339 1165 raft_consensus.cc:738] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.698432 1176 consensus_queue.cc:260] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.698573 1165 consensus_queue.cc:260] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.698851 1176 ts_tablet_manager.cc:1428] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 20:52:51.699090 1165 ts_tablet_manager.cc:1428] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:51.699121 1176 tablet_bootstrap.cc:492] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.701407 1176 tablet_bootstrap.cc:654] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.703895 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet cba68160be6145ddbbd2019c9b5f0076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.705448 1165 tablet_bootstrap.cc:492] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:51.705988 1166 ts_tablet_manager.cc:1918] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.706080 1166 log.cc:1198] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2a89a0b47d10414d9b6a9123af1f4ab3
I20250109 20:52:51.706400 1166 ts_tablet_manager.cc:1939] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:51.706971 1165 tablet_bootstrap.cc:654] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.707229 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2a89a0b47d10414d9b6a9123af1f4ab3 (table default.Table_ef823ba5464d429db5aa6b8c3ccc1934 [id=70a6ff8c791e41038b082b9004740857]) successfully deleted
I20250109 20:52:51.710606 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6716e7428ce240fca7ca5b02f564a5df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.715899 1702 raft_consensus.cc:491] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.716100 1702 raft_consensus.cc:513] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.716868 1702 leader_election.cc:290] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:51.717610 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f494150713b4dde9399d53d57567983" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:51.717756 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f494150713b4dde9399d53d57567983" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:51.718122 990 leader_election.cc:343] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:51.718245 989 leader_election.cc:343] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.718379 989 leader_election.cc:304] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:51.718797 1702 raft_consensus.cc:2743] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.719103 1160 ts_tablet_manager.cc:1918] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.719269 1160 log.cc:1198] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ffd6ce78438841daa663b6344f062019
I20250109 20:52:51.719645 1160 ts_tablet_manager.cc:1939] T ffd6ce78438841daa663b6344f062019 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:51.720554 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ffd6ce78438841daa663b6344f062019 (table default.Table_d8369c071899441a9b05f8910c9743c2 [id=b939987c4a0f426b846403a8579622c7]) successfully deleted
W20250109 20:52:51.721417 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 22f6ec8f3d4f4d37a11f15173449155c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22f6ec8f3d4f4d37a11f15173449155c already in progress: creating tablet
I20250109 20:52:51.723634 1166 tablet_replica.cc:331] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:51.723902 1166 raft_consensus.cc:2238] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.724148 1166 raft_consensus.cc:2267] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.724737 1166 ts_tablet_manager.cc:1905] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:51.725234 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9170b3a6b9bb446b8a474ae4df249409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9170b3a6b9bb446b8a474ae4df249409 already in progress: creating tablet
W20250109 20:52:51.725975 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6fb563bd6802473fafedd08bb10d15c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fb563bd6802473fafedd08bb10d15c1 already in progress: creating tablet
W20250109 20:52:51.726847 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 47726b58d6714cd1b46848a16a34ff37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47726b58d6714cd1b46848a16a34ff37 already in progress: creating tablet
W20250109 20:52:51.727677 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f2c83e709a8d4e078e67af58ec51beac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2c83e709a8d4e078e67af58ec51beac already in progress: creating tablet
I20250109 20:52:51.728222 1160 tablet_replica.cc:331] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:51.730105 1160 raft_consensus.cc:2238] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.730530 1160 raft_consensus.cc:2267] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.741886 1158 tablet_bootstrap.cc:492] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.742159 1158 ts_tablet_manager.cc:1397] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.004s
I20250109 20:52:51.742532 1160 ts_tablet_manager.cc:1905] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.745419 1158 raft_consensus.cc:357] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.746923 1158 raft_consensus.cc:383] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.747051 1158 raft_consensus.cc:738] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.747310 1158 consensus_queue.cc:260] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.747869 1158 ts_tablet_manager.cc:1428] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:52:51.748180 1158 tablet_bootstrap.cc:492] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:51.752398 1158 tablet_bootstrap.cc:654] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.757726 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2985f6de46ed417d87782a929b776d9a" } new_table_name: "default.Table_e6a87107149f421e93c2ef7e26400ef1" modify_external_catalogs: true
I20250109 20:52:51.759011 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 6716e7428ce240fca7ca5b02f564a5df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.759227 1157 tablet_replica.cc:331] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:51.759389 1157 raft_consensus.cc:2238] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.759517 1157 raft_consensus.cc:2267] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.760450 1157 ts_tablet_manager.cc:1905] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:51.765235 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33c7eef84a0c46de990aee95ada1e48d has no consensus state
I20250109 20:52:51.766805 1165 tablet_bootstrap.cc:492] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:51.766979 1165 ts_tablet_manager.cc:1397] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.003s
I20250109 20:52:51.767726 1165 raft_consensus.cc:357] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.767922 1165 raft_consensus.cc:383] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.767997 1165 raft_consensus.cc:738] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.768313 1165 consensus_queue.cc:260] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.768877 1165 ts_tablet_manager.cc:1428] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250109 20:52:51.769685 1158 tablet_bootstrap.cc:492] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.770509 1165 tablet_bootstrap.cc:492] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:51.772104 1165 tablet_bootstrap.cc:654] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.774876 1176 tablet_bootstrap.cc:492] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.775032 1176 ts_tablet_manager.cc:1397] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.001s
I20250109 20:52:51.775758 1176 raft_consensus.cc:357] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.775949 1176 raft_consensus.cc:383] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.776026 1176 raft_consensus.cc:738] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.776249 1176 consensus_queue.cc:260] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.776796 1176 ts_tablet_manager.cc:1428] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:51.778506 1176 tablet_bootstrap.cc:492] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.779799 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.780023 1176 tablet_bootstrap.cc:654] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.784745 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b153dbb4314c4300ac487674dda515eb" } new_table_name: "default.Table_f20230357dc241449b6b6dc5f5479105" modify_external_catalogs: true
I20250109 20:52:51.786710 450 master_service.cc:946] ReplaceTablet: received request to replace tablet a8da9781ab32429297d0c875c1971501 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:51.790511 1158 ts_tablet_manager.cc:1397] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.003s
I20250109 20:52:51.790917 1166 ts_tablet_manager.cc:1918] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.791133 1166 log.cc:1198] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/540531463c184b9abf226d7487939e60
I20250109 20:52:51.791556 1166 ts_tablet_manager.cc:1939] T 540531463c184b9abf226d7487939e60 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:51.792382 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b1891cbef8f4b0cae0c15d478db4860 consensus state has no leader
I20250109 20:52:51.792753 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 540531463c184b9abf226d7487939e60 (table default.Table_642aeae240da4ea3a0fca064e8489134 [id=59a36af89f8f4df0ace607f3190236ba]) successfully deleted
I20250109 20:52:51.792995 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0e564cfa1eb74fd8849dc79656731710" } modify_external_catalogs: true
I20250109 20:52:51.793042 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0191c74c02514d34a53c6db0d807f298" } modify_external_catalogs: true
W20250109 20:52:51.793567 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7df7405e93aa4718a62933ae56af9f42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7df7405e93aa4718a62933ae56af9f42 already in progress: creating tablet
I20250109 20:52:51.793565 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0191c74c02514d34a53c6db0d807f298" } modify_external_catalogs: true
I20250109 20:52:51.793560 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0e564cfa1eb74fd8849dc79656731710" } modify_external_catalogs: true
I20250109 20:52:51.793562 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_46b0126178104280ad6c27f0c3001a00" } modify_external_catalogs: true
I20250109 20:52:51.794075 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_46b0126178104280ad6c27f0c3001a00" } modify_external_catalogs: true
I20250109 20:52:51.794246 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_98192f0dc90240989c8b4c8c972d7751" } modify_external_catalogs: true
I20250109 20:52:51.794560 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_98192f0dc90240989c8b4c8c972d7751" } modify_external_catalogs: true
I20250109 20:52:51.794678 1157 ts_tablet_manager.cc:1918] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.796927 1157 log.cc:1198] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6716e7428ce240fca7ca5b02f564a5df
I20250109 20:52:51.798103 1157 ts_tablet_manager.cc:1939] T 6716e7428ce240fca7ca5b02f564a5df P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:51.796649 1158 raft_consensus.cc:357] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.798419 1158 raft_consensus.cc:383] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.799388 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6716e7428ce240fca7ca5b02f564a5df (table default.Table_bb480bfd701049cc8b814e0fe8b4c9ce [id=49d5df19d5124b71993545f365062601]) successfully deleted
I20250109 20:52:51.799592 1158 raft_consensus.cc:738] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.799921 1158 consensus_queue.cc:260] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.800477 1158 ts_tablet_manager.cc:1428] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250109 20:52:51.800964 1158 tablet_bootstrap.cc:492] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:51.801729 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 324d6ab550924d74809c491718313104
I20250109 20:52:51.806420 758 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c277dcc04d4acdbc116e41ad41d7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.808779 1158 tablet_bootstrap.cc:654] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.809027 1157 tablet_replica.cc:331] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:51.809284 1157 raft_consensus.cc:2238] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.809531 1157 raft_consensus.cc:2267] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.810307 1157 ts_tablet_manager.cc:1905] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.816031 1157 ts_tablet_manager.cc:1918] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.816388 1166 tablet_replica.cc:331] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:51.816622 1166 raft_consensus.cc:2238] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.816922 1166 raft_consensus.cc:2267] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.817443 1166 ts_tablet_manager.cc:1905] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.819909 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cd596614521a4a92a71b781b162f70b6
I20250109 20:52:51.821177 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596614521a4a92a71b781b162f70b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.821269 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2e1f70532d3045499f2c3ef554740247
I20250109 20:52:51.821499 893 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596614521a4a92a71b781b162f70b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.822579 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_bc43af02a045473cac4e0b21c2dfc318 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 1 got 0
I20250109 20:52:51.823244 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1f70532d3045499f2c3ef554740247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.823757 1160 ts_tablet_manager.cc:1918] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.823853 1160 log.cc:1198] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/7d981c7f13c3409f994bfdf0275e0116
I20250109 20:52:51.824193 1160 ts_tablet_manager.cc:1939] T 7d981c7f13c3409f994bfdf0275e0116 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:51.825747 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1f70532d3045499f2c3ef554740247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.822587 655 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2e1f70532d3045499f2c3ef554740247
I20250109 20:52:51.827034 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_bc43af02a045473cac4e0b21c2dfc318 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 1 got 0
I20250109 20:52:51.829929 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1f70532d3045499f2c3ef554740247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.830411 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet da7f67026b164e0faad128d34188afb5
I20250109 20:52:51.831302 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet da7f67026b164e0faad128d34188afb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.831624 893 tablet_service.cc:1514] Processing DeleteTablet for tablet da7f67026b164e0faad128d34188afb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.831897 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1f70532d3045499f2c3ef554740247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.833420 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_87926cb29c4d4a6483062503677f2798" } new_table_name: "default.Table_278a8479a4ac4de89f6895774e7eeeaa" modify_external_catalogs: true
I20250109 20:52:51.833777 893 tablet_service.cc:1467] Processing CreateTablet for tablet beee9825fa0240e88029bf32ace0e187 (DEFAULT_TABLE table=default.Table_bfad8a48292647e68716ec5438e5a6a7 [id=be2676e412c143c39fd150f5703f6f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.834218 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beee9825fa0240e88029bf32ace0e187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.834636 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9939ee8672474bc0bcd631684079d0c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.835165 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9939ee8672474bc0bcd631684079d0c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.837980 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ccc647e63d744a0cbfd48bdbd78f5f87" } modify_external_catalogs: true
I20250109 20:52:51.838160 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ccc647e63d744a0cbfd48bdbd78f5f87" } modify_external_catalogs: true
W20250109 20:52:51.839525 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c3af2a00ab04b41915cffc3eb9206bc consensus state has no leader
I20250109 20:52:51.841308 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c271524e66094f2e9f698e6e174de073"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.842422 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c271524e66094f2e9f698e6e174de073 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.845095 1165 tablet_bootstrap.cc:492] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:51.845257 1165 ts_tablet_manager.cc:1397] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20250109 20:52:51.845353 1030 tablet_service.cc:1467] Processing CreateTablet for tablet beee9825fa0240e88029bf32ace0e187 (DEFAULT_TABLE table=default.Table_bfad8a48292647e68716ec5438e5a6a7 [id=be2676e412c143c39fd150f5703f6f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.845790 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beee9825fa0240e88029bf32ace0e187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.846079 1165 raft_consensus.cc:357] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.846266 1165 raft_consensus.cc:383] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.846347 1165 raft_consensus.cc:738] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.846611 1165 consensus_queue.cc:260] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.847108 1165 ts_tablet_manager.cc:1428] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:51.847406 1165 tablet_bootstrap.cc:492] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:51.849656 1160 tablet_replica.cc:331] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:51.849870 1160 raft_consensus.cc:2238] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:51.850036 1160 raft_consensus.cc:2267] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:51.851194 1165 tablet_bootstrap.cc:654] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.857647 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 7d981c7f13c3409f994bfdf0275e0116 (table default.Table_2acc983c759e437cbce5b1a3a5e88cfe [id=b64c15100a8344a6bc35a896aec8270c]) successfully deleted
W20250109 20:52:51.857910 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7df7405e93aa4718a62933ae56af9f42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7df7405e93aa4718a62933ae56af9f42 already in progress: creating tablet
I20250109 20:52:51.861770 1702 raft_consensus.cc:491] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.861948 1702 raft_consensus.cc:513] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.862452 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9d13168ca13c400a8a2d9608293ff8c3
I20250109 20:52:51.862578 1702 leader_election.cc:290] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:51.863178 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d13168ca13c400a8a2d9608293ff8c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.864295 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d13168ca13c400a8a2d9608293ff8c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.864549 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3000b819a4cc4960909b98febd400631" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:51.865156 989 leader_election.cc:343] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.865583 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9feb9de9dc4d22bafc0f7707943113 with delete_type TABLET_DATA_DELETED (replaced by tablet e64207928ef141118e7fc8fe7bb2379b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.867676 1160 ts_tablet_manager.cc:1905] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:51.869316 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7cbb3a94bca245ed8531800592512f15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.869575 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_aab45704dc304088b76a3c55953aaef4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.869902 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cbb3a94bca245ed8531800592512f15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.870121 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aab45704dc304088b76a3c55953aaef4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.870033 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_06aabfaaf12e49bfa1196ae2f5850d21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.870471 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06aabfaaf12e49bfa1196ae2f5850d21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.870705 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fef953cfacde44d38f0e2dafb4114322" } new_table_name: "default.Table_8817ccacbbc345248314d8aa7d5cc7ba" modify_external_catalogs: true
I20250109 20:52:51.875556 893 tablet_service.cc:1467] Processing CreateTablet for tablet f7029146f2854a89999e87961db6b2e3 (DEFAULT_TABLE table=default.Table_937a1971308f4f759b74d375c3808729 [id=0ffb161aa1d0428a81a5576223c08858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.876426 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7029146f2854a89999e87961db6b2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.876482 1165 tablet_bootstrap.cc:492] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:51.876754 1657 raft_consensus.cc:491] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:51.876888 1657 raft_consensus.cc:513] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.877434 1657 leader_election.cc:290] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:51.878934 854 leader_election.cc:343] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2b9feb9de9dc4d22bafc0f7707943113
W20250109 20:52:51.879159 854 leader_election.cc:343] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 2b9feb9de9dc4d22bafc0f7707943113
I20250109 20:52:51.879253 854 leader_election.cc:304] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:51.879592 1657 raft_consensus.cc:2743] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.879938 1165 ts_tablet_manager.cc:1397] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20250109 20:52:51.880759 1165 raft_consensus.cc:357] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.881043 1165 raft_consensus.cc:383] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.881150 1165 raft_consensus.cc:738] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.881421 1165 consensus_queue.cc:260] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.883877 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_46839e2fe6664d1b9d90e1677fa0d012"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:51.884058 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_30b1ffb8aacc48c6883ed04b539160e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.884423 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46839e2fe6664d1b9d90e1677fa0d012 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.884503 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6464281d788c4197b1e88847c593282c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.884979 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6464281d788c4197b1e88847c593282c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:51.885352 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2769e5906c604bd5b14df79db765ec61 consensus state has no leader
W20250109 20:52:51.884423 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30b1ffb8aacc48c6883ed04b539160e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.885931 1030 tablet_service.cc:1467] Processing CreateTablet for tablet f7029146f2854a89999e87961db6b2e3 (DEFAULT_TABLE table=default.Table_937a1971308f4f759b74d375c3808729 [id=0ffb161aa1d0428a81a5576223c08858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.886278 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7029146f2854a89999e87961db6b2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.888594 1165 ts_tablet_manager.cc:1428] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250109 20:52:51.893491 1176 tablet_bootstrap.cc:492] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.893656 1176 ts_tablet_manager.cc:1397] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.000s
I20250109 20:52:51.893896 1165 tablet_bootstrap.cc:492] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:51.894551 1176 raft_consensus.cc:357] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.894816 1176 raft_consensus.cc:383] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.894901 1176 raft_consensus.cc:738] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.895148 1176 consensus_queue.cc:260] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.896201 1176 ts_tablet_manager.cc:1428] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:51.896484 1176 tablet_bootstrap.cc:492] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.897094 1165 tablet_bootstrap.cc:654] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.900976 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_10569a5c687543ae82b16452e4cb316f" } new_table_name: "default.Table_d7539ac3866842c996d4ced88e764877" modify_external_catalogs: true
I20250109 20:52:51.905058 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5fef44f0700046e0ba377025bb4d4c78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.905948 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fef44f0700046e0ba377025bb4d4c78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.906829 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b77c8d6adbf44375aff2782a83643255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.912144 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3beb193926bd4164a21137f32bf717ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:51.912786 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3beb193926bd4164a21137f32bf717ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:51.914494 1637 catalog_manager.cc:1260] Loaded cluster ID: 1852a89fe3204670aafdd57bab642604
I20250109 20:52:51.914655 1637 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: loading cluster ID for follower catalog manager: success
I20250109 20:52:51.919726 1160 ts_tablet_manager.cc:1918] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.921005 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 418e1bd414914b21a233c3099e2057da (DEFAULT_TABLE table=default.Table_5377c27f242f4aaf8ef9d0281640a856 [id=7521b80e658f4aaf9761df625056858c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.921439 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 418e1bd414914b21a233c3099e2057da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.922284 1176 tablet_bootstrap.cc:654] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:51.822053 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1f70532d3045499f2c3ef554740247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.825685 758 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596614521a4a92a71b781b162f70b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.828485 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1f70532d3045499f2c3ef554740247 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.830924 744 tablet_service.cc:1514] Processing DeleteTablet for tablet da7f67026b164e0faad128d34188afb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.832875 749 tablet_service.cc:1467] Processing CreateTablet for tablet beee9825fa0240e88029bf32ace0e187 (DEFAULT_TABLE table=default.Table_bfad8a48292647e68716ec5438e5a6a7 [id=be2676e412c143c39fd150f5703f6f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.925326 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beee9825fa0240e88029bf32ace0e187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.863266 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3000b819a4cc4960909b98febd400631" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:51.864073 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d13168ca13c400a8a2d9608293ff8c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.876515 754 tablet_service.cc:1467] Processing CreateTablet for tablet f7029146f2854a89999e87961db6b2e3 (DEFAULT_TABLE table=default.Table_937a1971308f4f759b74d375c3808729 [id=0ffb161aa1d0428a81a5576223c08858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.929440 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7029146f2854a89999e87961db6b2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.930569 756 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5609eaf86d4e8ea3d3da41243c927f (DEFAULT_TABLE table=default.Table_141b5c9b04d6457f8ebe492f7a7fb482 [id=2e25adc0595647c88f6cf27458078a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.930969 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5609eaf86d4e8ea3d3da41243c927f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:51.932906 990 leader_election.cc:343] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:51.933029 990 leader_election.cc:304] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:51.933463 1702 raft_consensus.cc:2743] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:51.917819 757 tablet_service.cc:1467] Processing CreateTablet for tablet 418e1bd414914b21a233c3099e2057da (DEFAULT_TABLE table=default.Table_5377c27f242f4aaf8ef9d0281640a856 [id=7521b80e658f4aaf9761df625056858c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.933928 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 418e1bd414914b21a233c3099e2057da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.938390 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5609eaf86d4e8ea3d3da41243c927f (DEFAULT_TABLE table=default.Table_141b5c9b04d6457f8ebe492f7a7fb482 [id=2e25adc0595647c88f6cf27458078a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.938828 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5609eaf86d4e8ea3d3da41243c927f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.945096 1157 log.cc:1198] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f7c277dcc04d4acdbc116e41ad41d7cf
I20250109 20:52:51.945487 1157 ts_tablet_manager.cc:1939] T f7c277dcc04d4acdbc116e41ad41d7cf P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:52:51.945899 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d1630e4438e4337ba026b41f630a480 consensus state has no leader
I20250109 20:52:51.946334 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f7c277dcc04d4acdbc116e41ad41d7cf (table default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]) successfully deleted
W20250109 20:52:51.946766 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2e1f70532d3045499f2c3ef554740247 because the tablet was not found. No further retry: Not found: Tablet not found: 2e1f70532d3045499f2c3ef554740247
W20250109 20:52:51.946988 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cd596614521a4a92a71b781b162f70b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd596614521a4a92a71b781b162f70b6 already in progress: creating tablet
W20250109 20:52:51.947220 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2e1f70532d3045499f2c3ef554740247 because the tablet was not found. No further retry: Not found: Tablet not found: 2e1f70532d3045499f2c3ef554740247
W20250109 20:52:51.947409 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet da7f67026b164e0faad128d34188afb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da7f67026b164e0faad128d34188afb5 already in progress: creating tablet
W20250109 20:52:51.947638 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9d13168ca13c400a8a2d9608293ff8c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d13168ca13c400a8a2d9608293ff8c3 already in progress: creating tablet
I20250109 20:52:51.952244 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9041a2e91f4753aa561965cc0e6280 (DEFAULT_TABLE table=default.Table_7193fd56447746c98aef50288388883c [id=2ac070c614204350995600047719af1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.952662 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9041a2e91f4753aa561965cc0e6280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.953505 446 master_service.cc:946] ReplaceTablet: received request to replace tablet bd62a3ae64a143998bb1172b3a97c69f from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:51.959470 1166 ts_tablet_manager.cc:1918] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:51.959718 1166 log.cc:1198] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6acca65ecee94f8cbe3f2b89e744aa6c
I20250109 20:52:51.960183 1166 ts_tablet_manager.cc:1939] T 6acca65ecee94f8cbe3f2b89e744aa6c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:51.925935 890 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e8ee4b86a04199865acd5f69cda384 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.926318 883 tablet_service.cc:1467] Processing CreateTablet for tablet 418e1bd414914b21a233c3099e2057da (DEFAULT_TABLE table=default.Table_5377c27f242f4aaf8ef9d0281640a856 [id=7521b80e658f4aaf9761df625056858c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.961848 1158 tablet_bootstrap.cc:492] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:51.962114 1158 ts_tablet_manager.cc:1397] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.161s user 0.007s sys 0.006s
I20250109 20:52:51.963137 890 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9041a2e91f4753aa561965cc0e6280 (DEFAULT_TABLE table=default.Table_7193fd56447746c98aef50288388883c [id=2ac070c614204350995600047719af1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.963171 1158 raft_consensus.cc:357] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.963421 1158 raft_consensus.cc:383] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.963511 1158 raft_consensus.cc:738] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.945919 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5609eaf86d4e8ea3d3da41243c927f (DEFAULT_TABLE table=default.Table_141b5c9b04d6457f8ebe492f7a7fb482 [id=2e25adc0595647c88f6cf27458078a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.963761 1158 consensus_queue.cc:260] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.964651 1160 log.cc:1198] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/6716e7428ce240fca7ca5b02f564a5df
I20250109 20:52:51.967442 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a9eb8655db18431ab35d434cfe59445c" } new_table_name: "default.Table_09933fbabe6e4a0885162da5f6e038db" modify_external_catalogs: true
I20250109 20:52:51.966956 749 tablet_service.cc:1467] Processing CreateTablet for tablet 2f53cef0adbb47d4840e4c11a6d0c8ac (DEFAULT_TABLE table=default.Table_ed658ec6109f45998eb6e916f76e8a79 [id=c8701071e1d6496686497a84816c186c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.969172 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bd62a3ae64a143998bb1172b3a97c69f
I20250109 20:52:51.969692 446 catalog_manager.cc:6596] ReplaceTablet: tablet bd62a3ae64a143998bb1172b3a97c69f deleted and replaced by tablet 575f3cb9f61143ab949fa85f47edcafb
I20250109 20:52:51.970835 897 tablet_service.cc:1514] Processing DeleteTablet for tablet bd62a3ae64a143998bb1172b3a97c69f with delete_type TABLET_DATA_DELETED (replaced by tablet 575f3cb9f61143ab949fa85f47edcafb) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:51.972188 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f53cef0adbb47d4840e4c11a6d0c8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.970885 754 tablet_service.cc:1514] Processing DeleteTablet for tablet bd62a3ae64a143998bb1172b3a97c69f with delete_type TABLET_DATA_DELETED (replaced by tablet 575f3cb9f61143ab949fa85f47edcafb) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:51.970033 1158 ts_tablet_manager.cc:1428] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20250109 20:52:51.977476 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a2efcc2a6ec4b109845d8087491ca9f consensus state has no leader
W20250109 20:52:51.978102 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet bd62a3ae64a143998bb1172b3a97c69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd62a3ae64a143998bb1172b3a97c69f already in progress: creating tablet
I20250109 20:52:51.976161 1160 ts_tablet_manager.cc:1939] T 6716e7428ce240fca7ca5b02f564a5df P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:51.960238 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9041a2e91f4753aa561965cc0e6280 (DEFAULT_TABLE table=default.Table_7193fd56447746c98aef50288388883c [id=2ac070c614204350995600047719af1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.971277 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet bd62a3ae64a143998bb1172b3a97c69f with delete_type TABLET_DATA_DELETED (replaced by tablet 575f3cb9f61143ab949fa85f47edcafb) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:51.979539 1176 tablet_bootstrap.cc:492] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:51.980741 897 tablet_service.cc:1467] Processing CreateTablet for tablet 2f53cef0adbb47d4840e4c11a6d0c8ac (DEFAULT_TABLE table=default.Table_ed658ec6109f45998eb6e916f76e8a79 [id=c8701071e1d6496686497a84816c186c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:51.980890 1176 ts_tablet_manager.cc:1397] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.000s
I20250109 20:52:51.981886 1176 raft_consensus.cc:357] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.982273 1176 raft_consensus.cc:383] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:51.982218 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9041a2e91f4753aa561965cc0e6280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.982411 1176 raft_consensus.cc:738] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:51.983543 1176 consensus_queue.cc:260] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:51.985966 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9041a2e91f4753aa561965cc0e6280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.986157 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f53cef0adbb47d4840e4c11a6d0c8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.988903 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 418e1bd414914b21a233c3099e2057da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.988886 1176 ts_tablet_manager.cc:1428] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250109 20:52:51.992334 1176 tablet_bootstrap.cc:492] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:51.999354 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5609eaf86d4e8ea3d3da41243c927f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:51.999706 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5505cf6536bf496e88f11dc1586941d0" } new_table_name: "default.Table_4c73eadb1a204cbdb9d845b96cd2c4b1" modify_external_catalogs: true
I20250109 20:52:52.007689 1158 tablet_bootstrap.cc:492] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.007999 754 tablet_service.cc:1467] Processing CreateTablet for tablet 65570e42a9bd45bcba0a82abd12a9356 (DEFAULT_TABLE table=default.Table_d773b9e15aff4f4fa339a5c6c2e70bd8 [id=426a2d55e76f4866b0f52ae174d1b643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.008512 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65570e42a9bd45bcba0a82abd12a9356. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.008690 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c344526523b540b7ae0c1d1853a60c23" } modify_external_catalogs: true
I20250109 20:52:52.008905 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c344526523b540b7ae0c1d1853a60c23" } modify_external_catalogs: true
I20250109 20:52:52.009630 1160 tablet_replica.cc:331] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:52.009827 1160 raft_consensus.cc:2238] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.010003 1160 raft_consensus.cc:2267] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.010460 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 6716e7428ce240fca7ca5b02f564a5df (table default.Table_bb480bfd701049cc8b814e0fe8b4c9ce [id=49d5df19d5124b71993545f365062601]) successfully deleted
I20250109 20:52:52.011610 883 tablet_service.cc:1467] Processing CreateTablet for tablet 65570e42a9bd45bcba0a82abd12a9356 (DEFAULT_TABLE table=default.Table_d773b9e15aff4f4fa339a5c6c2e70bd8 [id=426a2d55e76f4866b0f52ae174d1b643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.011930 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65570e42a9bd45bcba0a82abd12a9356. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.017127 754 tablet_service.cc:1467] Processing CreateTablet for tablet 274a58191d00475e8da9ff72dd33524f (DEFAULT_TABLE table=default.Table_6dfedfbc9bba4dc8a4fb00f68b3ba689 [id=d93249e4fcc34febb8b284a21dc33364]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.017287 1160 ts_tablet_manager.cc:1905] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.017580 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274a58191d00475e8da9ff72dd33524f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:52.018728 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44af81df98ae4b05927dab7b4246de9f consensus state has no leader
I20250109 20:52:52.022352 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7ad564a8e67d49bb8a46048a1c78ddd9" } modify_external_catalogs: true
I20250109 20:52:52.022588 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7ad564a8e67d49bb8a46048a1c78ddd9" } modify_external_catalogs: true
I20250109 20:52:52.023290 757 tablet_service.cc:1467] Processing CreateTablet for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b (DEFAULT_TABLE table=default.Table_e0cf5166142546989eb825944cae0691 [id=ebdd987074c643c6a5de34374b5002d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.023626 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.023959 756 tablet_service.cc:1467] Processing CreateTablet for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 (DEFAULT_TABLE table=default.Table_47a796218c0848d9990864cc20f7ae05 [id=f005df4cb1524ed2945b5589597e15c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.024379 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1faf5d79a9b43ffafc9c34a5937fcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.024464 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_141b5c9b04d6457f8ebe492f7a7fb482" } modify_external_catalogs: true
I20250109 20:52:52.024616 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_141b5c9b04d6457f8ebe492f7a7fb482" } modify_external_catalogs: true
I20250109 20:52:52.029143 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f2383d9770834b1aae290308b4185299
I20250109 20:52:52.029902 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f2383d9770834b1aae290308b4185299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.030552 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f2383d9770834b1aae290308b4185299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2383d9770834b1aae290308b4185299 already in progress: creating tablet
I20250109 20:52:52.031672 888 tablet_service.cc:1514] Processing DeleteTablet for tablet f2383d9770834b1aae290308b4185299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.032024 890 tablet_service.cc:1467] Processing CreateTablet for tablet 274a58191d00475e8da9ff72dd33524f (DEFAULT_TABLE table=default.Table_6dfedfbc9bba4dc8a4fb00f68b3ba689 [id=d93249e4fcc34febb8b284a21dc33364]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.032341 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274a58191d00475e8da9ff72dd33524f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.032630 1678 raft_consensus.cc:491] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.032821 1678 raft_consensus.cc:513] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.033514 1678 leader_election.cc:290] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:52.034307 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b06e8616df461d89c58d414fc0cfbd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:52.034829 712 leader_election.cc:343] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.036401 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 2f53cef0adbb47d4840e4c11a6d0c8ac (DEFAULT_TABLE table=default.Table_ed658ec6109f45998eb6e916f76e8a79 [id=c8701071e1d6496686497a84816c186c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.036963 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 274a58191d00475e8da9ff72dd33524f (DEFAULT_TABLE table=default.Table_6dfedfbc9bba4dc8a4fb00f68b3ba689 [id=d93249e4fcc34febb8b284a21dc33364]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.037899 1033 tablet_service.cc:1467] Processing CreateTablet for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b (DEFAULT_TABLE table=default.Table_e0cf5166142546989eb825944cae0691 [id=ebdd987074c643c6a5de34374b5002d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.038476 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet f2383d9770834b1aae290308b4185299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.039101 897 tablet_service.cc:1467] Processing CreateTablet for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b (DEFAULT_TABLE table=default.Table_e0cf5166142546989eb825944cae0691 [id=ebdd987074c643c6a5de34374b5002d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.039243 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 65570e42a9bd45bcba0a82abd12a9356 (DEFAULT_TABLE table=default.Table_d773b9e15aff4f4fa339a5c6c2e70bd8 [id=426a2d55e76f4866b0f52ae174d1b643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.039497 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.039620 1702 raft_consensus.cc:491] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.039680 447 master_service.cc:946] ReplaceTablet: received request to replace tablet deedac68322e460b8d62adbe849de3f2 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.039844 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.040132 1702 raft_consensus.cc:513] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.040896 1702 leader_election.cc:290] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:52.041077 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b06e8616df461d89c58d414fc0cfbd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:52.041302 1053 raft_consensus.cc:2463] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:52.041997 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc2c7b12f945ea8307a187e1553226" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.042305 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_68df6c0c372443ba892b057eacff3ffa" } modify_external_catalogs: true
I20250109 20:52:52.042439 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_68df6c0c372443ba892b057eacff3ffa" } modify_external_catalogs: true
I20250109 20:52:52.042724 713 leader_election.cc:304] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:52.043510 990 leader_election.cc:343] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.043987 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f53cef0adbb47d4840e4c11a6d0c8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.048683 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274a58191d00475e8da9ff72dd33524f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.048988 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65570e42a9bd45bcba0a82abd12a9356. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.049801 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6acca65ecee94f8cbe3f2b89e744aa6c (table default.Table_4792b8689a12402f9946e87ad47dcbbb [id=ec2d9d93307747a7b752255f2008500e]) successfully deleted
I20250109 20:52:52.050308 1678 raft_consensus.cc:2798] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:52.050443 1678 raft_consensus.cc:491] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:52.050534 1678 raft_consensus.cc:3054] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.052858 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.056931 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc2c7b12f945ea8307a187e1553226" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:52.057670 989 leader_election.cc:343] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.057862 989 leader_election.cc:304] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:52.058435 1658 raft_consensus.cc:491] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.058485 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d9f18f244a344272bba2bc085686b61d
I20250109 20:52:52.058595 1658 raft_consensus.cc:513] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.059345 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2f5609eaf86d4e8ea3d3da41243c927f
I20250109 20:52:52.059334 1658 leader_election.cc:290] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:52.059506 1658 raft_consensus.cc:2743] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:52.060326 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba68160be6145ddbbd2019c9b5f0076" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.060760 1166 tablet_replica.cc:331] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:52.060952 990 leader_election.cc:343] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.062589 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f18f244a344272bba2bc085686b61d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.062871 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5609eaf86d4e8ea3d3da41243c927f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.063257 1158 tablet_bootstrap.cc:654] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.068115 1678 raft_consensus.cc:513] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.068223 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_60c85531239c48349f1931cbcf6b2977" } modify_external_catalogs: true
I20250109 20:52:52.068449 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_60c85531239c48349f1931cbcf6b2977" } modify_external_catalogs: true
I20250109 20:52:52.068699 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 (DEFAULT_TABLE table=default.Table_47a796218c0848d9990864cc20f7ae05 [id=f005df4cb1524ed2945b5589597e15c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.069304 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1faf5d79a9b43ffafc9c34a5937fcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.069298 1678 leader_election.cc:290] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:52.070477 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_067ec85f2e5e4ac1bd60cba79be3ac24" } new_table_name: "default.Table_f91af69e24f743448574fff0b1e15773" modify_external_catalogs: true
I20250109 20:52:52.073227 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet deedac68322e460b8d62adbe849de3f2
I20250109 20:52:52.072877 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5609eaf86d4e8ea3d3da41243c927f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.073624 447 catalog_manager.cc:6596] ReplaceTablet: tablet deedac68322e460b8d62adbe849de3f2 deleted and replaced by tablet be05de223c3946c6b8a4899cf696fe26
I20250109 20:52:52.074007 757 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f18f244a344272bba2bc085686b61d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.074203 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2f5609eaf86d4e8ea3d3da41243c927f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f5609eaf86d4e8ea3d3da41243c927f already in progress: creating tablet
W20250109 20:52:52.074997 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d9f18f244a344272bba2bc085686b61d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f18f244a344272bba2bc085686b61d already in progress: creating tablet
I20250109 20:52:52.075505 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet deedac68322e460b8d62adbe849de3f2 with delete_type TABLET_DATA_DELETED (replaced by tablet be05de223c3946c6b8a4899cf696fe26) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.075783 1176 tablet_bootstrap.cc:654] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.076607 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet c71e4f79179d4d35956fbab05774982b
I20250109 20:52:52.077442 757 tablet_service.cc:1514] Processing DeleteTablet for tablet deedac68322e460b8d62adbe849de3f2 with delete_type TABLET_DATA_DELETED (replaced by tablet be05de223c3946c6b8a4899cf696fe26) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.078147 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet deedac68322e460b8d62adbe849de3f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deedac68322e460b8d62adbe849de3f2 already in progress: creating tablet
I20250109 20:52:52.080127 756 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0cc2be25940d08fad42a8c8d8a07b (DEFAULT_TABLE table=default.Table_189b7740a7c34ecbbec972fa47180efe [id=902f745269374ba6a0c4ecbeeab62bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.081928 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f0cc2be25940d08fad42a8c8d8a07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.082717 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b06e8616df461d89c58d414fc0cfbd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:52.083348 712 leader_election.cc:343] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.060978 1166 raft_consensus.cc:2238] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.085834 1166 raft_consensus.cc:2267] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.086498 1166 ts_tablet_manager.cc:1905] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.087867 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba68160be6145ddbbd2019c9b5f0076" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:52.088179 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f18f244a344272bba2bc085686b61d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.088399 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5609eaf86d4e8ea3d3da41243c927f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.088825 898 tablet_service.cc:1514] Processing DeleteTablet for tablet deedac68322e460b8d62adbe849de3f2 with delete_type TABLET_DATA_DELETED (replaced by tablet be05de223c3946c6b8a4899cf696fe26) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.089017 880 tablet_service.cc:1467] Processing CreateTablet for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 (DEFAULT_TABLE table=default.Table_47a796218c0848d9990864cc20f7ae05 [id=f005df4cb1524ed2945b5589597e15c4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:52.089594 989 leader_election.cc:343] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.090242 989 leader_election.cc:304] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:52.090868 1702 raft_consensus.cc:2743] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:52.095077 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1faf5d79a9b43ffafc9c34a5937fcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.096458 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 12e97ff530df46ae80eb1b5615a55a3f
I20250109 20:52:52.098676 744 tablet_service.cc:1467] Processing CreateTablet for tablet fd92f4c7abea454c8d68c1f5023d5727 (DEFAULT_TABLE table=default.Table_cf3f4f1f920642a396c170041d629601 [id=f005df9b45fb4419805df721df0164b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.099153 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd92f4c7abea454c8d68c1f5023d5727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.099956 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_90e645af6f5e4109a7f76ed0ae34b4ed" } new_table_name: "default.Table_19aea38f8d4547399b10df16cdb1e54e" modify_external_catalogs: true
I20250109 20:52:52.100991 896 tablet_service.cc:1467] Processing CreateTablet for tablet fd92f4c7abea454c8d68c1f5023d5727 (DEFAULT_TABLE table=default.Table_cf3f4f1f920642a396c170041d629601 [id=f005df9b45fb4419805df721df0164b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.101012 739 tablet_service.cc:1467] Processing CreateTablet for tablet 9382be89bcf1452ab33f3deede173d26 (DEFAULT_TABLE table=default.Table_067ec85f2e5e4ac1bd60cba79be3ac24 [id=2a8066a7d1c5476c94dc75159214e033]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.101406 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9382be89bcf1452ab33f3deede173d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.103744 1158 tablet_bootstrap.cc:492] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.103901 1158 ts_tablet_manager.cc:1397] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.129s user 0.000s sys 0.004s
I20250109 20:52:52.104746 1158 raft_consensus.cc:357] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.104995 1158 raft_consensus.cc:383] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.105079 1158 raft_consensus.cc:738] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.105301 1158 consensus_queue.cc:260] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.105753 1158 ts_tablet_manager.cc:1428] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:52.106283 758 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1f70532d3045499f2c3ef554740247 (DEFAULT_TABLE table=default.Table_98192f0dc90240989c8b4c8c972d7751 [id=d7e1cfb33cfd47539ba66a6faf359b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.106683 1158 tablet_bootstrap.cc:492] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.109851 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4cf7db6076db4bc4a5b7527f7b0a41cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.110404 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cf7db6076db4bc4a5b7527f7b0a41cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.110625 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e1f70532d3045499f2c3ef554740247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.111711 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 617029392e994f44a60c71ba2e340c57 (DEFAULT_TABLE table=default.Table_eb2e935fd7fb43c0ba59c503c575d803 [id=2fb050da26d248e4a40ce27d5e50667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.112155 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 617029392e994f44a60c71ba2e340c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.117058 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1f70532d3045499f2c3ef554740247 (DEFAULT_TABLE table=default.Table_98192f0dc90240989c8b4c8c972d7751 [id=d7e1cfb33cfd47539ba66a6faf359b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.117501 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e1f70532d3045499f2c3ef554740247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.121132 1033 tablet_service.cc:1467] Processing CreateTablet for tablet fd92f4c7abea454c8d68c1f5023d5727 (DEFAULT_TABLE table=default.Table_cf3f4f1f920642a396c170041d629601 [id=f005df9b45fb4419805df721df0164b7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:52.121343 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9382be89bcf1452ab33f3deede173d26 consensus state has no leader
I20250109 20:52:52.123914 887 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1f70532d3045499f2c3ef554740247 (DEFAULT_TABLE table=default.Table_98192f0dc90240989c8b4c8c972d7751 [id=d7e1cfb33cfd47539ba66a6faf359b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.125814 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e97ff530df46ae80eb1b5615a55a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.126598 898 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0cc2be25940d08fad42a8c8d8a07b (DEFAULT_TABLE table=default.Table_189b7740a7c34ecbbec972fa47180efe [id=902f745269374ba6a0c4ecbeeab62bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.126894 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd92f4c7abea454c8d68c1f5023d5727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.127830 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e1f70532d3045499f2c3ef554740247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.128449 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f0cc2be25940d08fad42a8c8d8a07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.128342 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b43e8c35fcdf4973ada77b9c4ad44ba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:52.128542 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c57d7e4c6798461280d6ff32c28b05f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.129050 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b43e8c35fcdf4973ada77b9c4ad44ba0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:52.129101 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c57d7e4c6798461280d6ff32c28b05f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.130319 895 tablet_service.cc:1467] Processing CreateTablet for tablet 9382be89bcf1452ab33f3deede173d26 (DEFAULT_TABLE table=default.Table_067ec85f2e5e4ac1bd60cba79be3ac24 [id=2a8066a7d1c5476c94dc75159214e033]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.130671 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6e96bb9287564f70abcd0b9f9865e77f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:52.130829 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f2e044c26f7c4b9d985304d701432239"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.131165 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e96bb9287564f70abcd0b9f9865e77f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:52.131217 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2e044c26f7c4b9d985304d701432239 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.133247 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd92f4c7abea454c8d68c1f5023d5727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.133566 756 tablet_service.cc:1467] Processing CreateTablet for tablet 617029392e994f44a60c71ba2e340c57 (DEFAULT_TABLE table=default.Table_eb2e935fd7fb43c0ba59c503c575d803 [id=2fb050da26d248e4a40ce27d5e50667f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:52.133858 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc54e97e91c54afea098624bd85fcc6e consensus state has no leader
I20250109 20:52:52.134130 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 617029392e994f44a60c71ba2e340c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.138191 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 9382be89bcf1452ab33f3deede173d26 (DEFAULT_TABLE table=default.Table_067ec85f2e5e4ac1bd60cba79be3ac24 [id=2a8066a7d1c5476c94dc75159214e033]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.138471 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e97ff530df46ae80eb1b5615a55a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.139266 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 12e97ff530df46ae80eb1b5615a55a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e97ff530df46ae80eb1b5615a55a3f already in progress: creating tablet
I20250109 20:52:52.143523 757 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1d5b0210134ce1a33b5e7603998593 (DEFAULT_TABLE table=default.Table_e1503bbd879343928760e2b8a1192c3a [id=37712738e1cc4a2687a492914a01cd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.143949 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1d5b0210134ce1a33b5e7603998593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.144503 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3b054fb299c84cf49c242e3ccb44daf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.145054 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b054fb299c84cf49c242e3ccb44daf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.146584 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_169be780c136447db26907a86d29867a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.147094 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_169be780c136447db26907a86d29867a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.149372 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_38191599779143059082c1ed7b3ccb76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:52.149536 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c9b2244c5a944c76a626d87d4ce80e24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:52.149698 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d664d9d1e75b4d728aadaa5ef1103542"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.149852 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38191599779143059082c1ed7b3ccb76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:52.150013 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9b2244c5a944c76a626d87d4ce80e24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:52.150151 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d664d9d1e75b4d728aadaa5ef1103542 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.150556 1702 raft_consensus.cc:491] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.151018 1702 raft_consensus.cc:513] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.151098 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_86ed54bc2bc046709614f244faefef48" } new_table_name: "default.Table_142271ba6e494ddd876859927b650918" modify_external_catalogs: true
I20250109 20:52:52.151641 1702 leader_election.cc:290] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:52.152284 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb563bd6802473fafedd08bb10d15c1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.152488 1158 tablet_bootstrap.cc:654] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:52.153832 990 leader_election.cc:343] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.154014 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e97ff530df46ae80eb1b5615a55a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.160774 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9382be89bcf1452ab33f3deede173d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.161705 885 tablet_service.cc:1467] Processing CreateTablet for tablet 617029392e994f44a60c71ba2e340c57 (DEFAULT_TABLE table=default.Table_eb2e935fd7fb43c0ba59c503c575d803 [id=2fb050da26d248e4a40ce27d5e50667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.162180 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 617029392e994f44a60c71ba2e340c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.163761 893 tablet_service.cc:1467] Processing CreateTablet for tablet 687db3e734af47339c2546970a71a539 (DEFAULT_TABLE table=default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.164119 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687db3e734af47339c2546970a71a539. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.164822 1702 consensus_queue.cc:1035] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:52.164682 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb563bd6802473fafedd08bb10d15c1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:52.165771 989 leader_election.cc:343] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.165904 989 leader_election.cc:304] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:52.166322 1702 raft_consensus.cc:2743] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:52.167006 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0cc2be25940d08fad42a8c8d8a07b (DEFAULT_TABLE table=default.Table_189b7740a7c34ecbbec972fa47180efe [id=902f745269374ba6a0c4ecbeeab62bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.167421 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9382be89bcf1452ab33f3deede173d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.177446 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 17940f311b0440a59a42cc57f3a0bc4c (DEFAULT_TABLE table=default.Table_bc4cb5b8559b475fb49723c73456ce40 [id=de1c3ba44e5e4f3984720e5d910e9314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.177976 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17940f311b0440a59a42cc57f3a0bc4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.179797 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 687db3e734af47339c2546970a71a539 (DEFAULT_TABLE table=default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.180253 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687db3e734af47339c2546970a71a539. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.181337 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f0cc2be25940d08fad42a8c8d8a07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.184774 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1d5b0210134ce1a33b5e7603998593 (DEFAULT_TABLE table=default.Table_e1503bbd879343928760e2b8a1192c3a [id=37712738e1cc4a2687a492914a01cd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.185230 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1d5b0210134ce1a33b5e7603998593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.160557 888 tablet_service.cc:1467] Processing CreateTablet for tablet 17940f311b0440a59a42cc57f3a0bc4c (DEFAULT_TABLE table=default.Table_bc4cb5b8559b475fb49723c73456ce40 [id=de1c3ba44e5e4f3984720e5d910e9314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.193740 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17940f311b0440a59a42cc57f3a0bc4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.197826 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e446345a181499d86750bfaca472d62" } new_table_name: "default.Table_e4aa837977f646d1962fa4a45ea438b2" modify_external_catalogs: true
I20250109 20:52:52.203406 1702 raft_consensus.cc:491] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.203562 1702 raft_consensus.cc:513] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.204245 1702 leader_election.cc:290] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:52.205121 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77c8d6adbf44375aff2782a83643255" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:52.205631 989 leader_election.cc:343] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.206914 452 master_service.cc:946] ReplaceTablet: received request to replace tablet feb9b0f722eb44b0acd5cb3bbb32c527 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.207391 897 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1d5b0210134ce1a33b5e7603998593 (DEFAULT_TABLE table=default.Table_e1503bbd879343928760e2b8a1192c3a [id=37712738e1cc4a2687a492914a01cd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.207608 1176 tablet_bootstrap.cc:492] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:52.207746 1176 ts_tablet_manager.cc:1397] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.215s user 0.004s sys 0.000s
I20250109 20:52:52.207791 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1d5b0210134ce1a33b5e7603998593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.208616 1176 raft_consensus.cc:357] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.214502 1160 ts_tablet_manager.cc:1918] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.214636 1160 log.cc:1198] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f7c277dcc04d4acdbc116e41ad41d7cf
I20250109 20:52:52.215044 1160 ts_tablet_manager.cc:1939] T f7c277dcc04d4acdbc116e41ad41d7cf P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:52.216431 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f7c277dcc04d4acdbc116e41ad41d7cf (table default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]) successfully deleted
I20250109 20:52:52.208850 1176 raft_consensus.cc:383] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.221684 1176 raft_consensus.cc:738] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.221962 1176 consensus_queue.cc:260] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.222222 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9c4c55aacaeb4e3faa678a177e936bbf" } new_table_name: "default.Table_604867c71c1740bf93813e9a6c0cb268" modify_external_catalogs: true
I20250109 20:52:52.222501 1176 ts_tablet_manager.cc:1428] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250109 20:52:52.224475 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 686ed5f7d5524c54a8debe1bc610bd25 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.224859 1176 tablet_bootstrap.cc:492] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:52.227279 1637 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: acquiring CA information for follower catalog manager: success
I20250109 20:52:52.240993 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7f957a53ae794811bf294551bf3f6e00" } modify_external_catalogs: true
I20250109 20:52:52.241168 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7f957a53ae794811bf294551bf3f6e00" } modify_external_catalogs: true
W20250109 20:52:52.244012 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62caa775ef1241f68c0323a2d228c8cd consensus state has no leader
W20250109 20:52:52.245115 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2420b285905342b3820f9549c8f0c4ad consensus state has no leader
I20250109 20:52:52.247748 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
I20250109 20:52:52.248092 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_b19d3251bcdf41eab27c97e89366d1c7 [id=50837eb593734f66a03e29d8edbc63a4]). Expected version 1 got 0
I20250109 20:52:52.248306 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:52.253213 1637 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:52:52.253360 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet feb9b0f722eb44b0acd5cb3bbb32c527
I20250109 20:52:52.253767 452 catalog_manager.cc:6596] ReplaceTablet: tablet feb9b0f722eb44b0acd5cb3bbb32c527 deleted and replaced by tablet d4b939cdea1c4719b9b3918aabbdfc0e
I20250109 20:52:52.255764 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet feb9b0f722eb44b0acd5cb3bbb32c527 with delete_type TABLET_DATA_DELETED (replaced by tablet d4b939cdea1c4719b9b3918aabbdfc0e) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.256269 1160 tablet_replica.cc:331] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:52.256525 1160 raft_consensus.cc:2238] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.256734 1160 raft_consensus.cc:2267] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.257678 1165 tablet_bootstrap.cc:492] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:52.257822 1165 ts_tablet_manager.cc:1397] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.364s user 0.063s sys 0.085s
I20250109 20:52:52.258491 1176 tablet_bootstrap.cc:654] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.258596 1165 raft_consensus.cc:357] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.258826 1165 raft_consensus.cc:383] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.258916 1165 raft_consensus.cc:738] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.259136 1165 consensus_queue.cc:260] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.259634 1165 ts_tablet_manager.cc:1428] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:52.259864 1165 tablet_bootstrap.cc:492] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:52.261322 1166 ts_tablet_manager.cc:1918] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.262022 1165 tablet_bootstrap.cc:654] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:52.262729 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8532291b4d0482690f3eed45c0f1f5b consensus state has no leader
I20250109 20:52:52.264823 1160 ts_tablet_manager.cc:1905] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.266924 1166 log.cc:1198] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/3000b819a4cc4960909b98febd400631
I20250109 20:52:52.267270 1166 ts_tablet_manager.cc:1939] T 3000b819a4cc4960909b98febd400631 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:52.267521 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 75cdf5b737504d4e8f598cd2b265192e (table default.Table_814844402c5e4defbb723b986171a2a2 [id=a4138fd6ff5246d582fc0d75b890c46f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 75cdf5b737504d4e8f598cd2b265192e consensus state has no leader
W20250109 20:52:52.267910 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cd596614521a4a92a71b781b162f70b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd596614521a4a92a71b781b162f70b6 already in progress: creating tablet
I20250109 20:52:52.268115 1166 tablet_replica.cc:331] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:52.268148 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2e1f70532d3045499f2c3ef554740247 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e1f70532d3045499f2c3ef554740247 already in progress: creating tablet
I20250109 20:52:52.268288 1166 raft_consensus.cc:2238] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.268436 1166 raft_consensus.cc:2267] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:52.268769 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2e1f70532d3045499f2c3ef554740247 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e1f70532d3045499f2c3ef554740247 already in progress: creating tablet
I20250109 20:52:52.268962 1166 ts_tablet_manager.cc:1905] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:52.269614 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet da7f67026b164e0faad128d34188afb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da7f67026b164e0faad128d34188afb5 already in progress: creating tablet
W20250109 20:52:52.269851 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9d13168ca13c400a8a2d9608293ff8c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d13168ca13c400a8a2d9608293ff8c3 already in progress: creating tablet
I20250109 20:52:52.270046 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_df97156287614f04b00e64a3ecfb64ce" } modify_external_catalogs: true
I20250109 20:52:52.270237 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_df97156287614f04b00e64a3ecfb64ce" } modify_external_catalogs: true
I20250109 20:52:52.271292 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 3000b819a4cc4960909b98febd400631 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]) successfully deleted
I20250109 20:52:52.272006 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c9b2244c5a944c76a626d87d4ce80e24" } modify_external_catalogs: true
I20250109 20:52:52.272161 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c9b2244c5a944c76a626d87d4ce80e24" } modify_external_catalogs: true
I20250109 20:52:52.272609 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 33c7eef84a0c46de990aee95ada1e48d (DEFAULT_TABLE table=default.Table_e6a87107149f421e93c2ef7e26400ef1 [id=7878e437152b4d90b49864428cd94ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.274192 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 37ee437300414eccba9926e824f72f27
I20250109 20:52:52.277915 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 67be9533983443cbb8fc52031216a973 (DEFAULT_TABLE table=default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.278767 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c7eef84a0c46de990aee95ada1e48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.279321 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67be9533983443cbb8fc52031216a973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.279542 757 tablet_service.cc:1467] Processing CreateTablet for tablet 687db3e734af47339c2546970a71a539 (DEFAULT_TABLE table=default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.280331 1702 consensus_queue.cc:1035] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:52.282255 893 tablet_service.cc:1514] Processing DeleteTablet for tablet feb9b0f722eb44b0acd5cb3bbb32c527 with delete_type TABLET_DATA_DELETED (replaced by tablet d4b939cdea1c4719b9b3918aabbdfc0e) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.282320 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77c8d6adbf44375aff2782a83643255" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.282821 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687db3e734af47339c2546970a71a539. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.284545 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 31de9638669040db9674d3aa7df32103
I20250109 20:52:52.285861 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_54acd558d6f04df09ecc0669cabfe878" } modify_external_catalogs: true
I20250109 20:52:52.286048 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_54acd558d6f04df09ecc0669cabfe878" } modify_external_catalogs: true
W20250109 20:52:52.288175 990 leader_election.cc:343] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.289501 749 tablet_service.cc:1467] Processing CreateTablet for tablet 17940f311b0440a59a42cc57f3a0bc4c (DEFAULT_TABLE table=default.Table_bc4cb5b8559b475fb49723c73456ce40 [id=de1c3ba44e5e4f3984720e5d910e9314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.289857 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17940f311b0440a59a42cc57f3a0bc4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.290781 758 tablet_service.cc:1467] Processing CreateTablet for tablet 67be9533983443cbb8fc52031216a973 (DEFAULT_TABLE table=default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.292256 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ee437300414eccba9926e824f72f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.298663 990 leader_election.cc:304] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:52.299234 751 tablet_service.cc:1467] Processing CreateTablet for tablet b8532291b4d0482690f3eed45c0f1f5b (DEFAULT_TABLE table=default.Table_604867c71c1740bf93813e9a6c0cb268 [id=29ed39cbeeb7486993274f97a291596b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.299423 897 tablet_service.cc:1467] Processing CreateTablet for tablet 67be9533983443cbb8fc52031216a973 (DEFAULT_TABLE table=default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.299851 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67be9533983443cbb8fc52031216a973. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:52.300915 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 37ee437300414eccba9926e824f72f27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ee437300414eccba9926e824f72f27 already in progress: creating tablet
I20250109 20:52:52.301265 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_45d77a3e61d14f468ad3e271195a1cd9" } modify_external_catalogs: true
I20250109 20:52:52.301443 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_45d77a3e61d14f468ad3e271195a1cd9" } modify_external_catalogs: true
I20250109 20:52:52.301995 1027 tablet_service.cc:1467] Processing CreateTablet for tablet b8532291b4d0482690f3eed45c0f1f5b (DEFAULT_TABLE table=default.Table_604867c71c1740bf93813e9a6c0cb268 [id=29ed39cbeeb7486993274f97a291596b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.301450 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 686ed5f7d5524c54a8debe1bc610bd25 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.302675 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 829692ee165c4546912fd32ea58e4356
I20250109 20:52:52.301067 1702 raft_consensus.cc:2743] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:52.303588 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ee437300414eccba9926e824f72f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.307612 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cbef1d27c5394fc1b6354c5d6f3a8f1f
I20250109 20:52:52.314692 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e716270ffcf44039b075023fd0c2d2a3
I20250109 20:52:52.319595 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ee437300414eccba9926e824f72f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.319950 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 31de9638669040db9674d3aa7df32103 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.320472 880 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0bfb44367a4572a9507af0448af846 (DEFAULT_TABLE table=default.Table_1340076b6d8146429e90b485b3b58728 [id=141a963a55d14eba985268a2767410d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.320860 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a0bfb44367a4572a9507af0448af846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.321383 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet e716270ffcf44039b075023fd0c2d2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.323447 896 tablet_service.cc:1514] Processing DeleteTablet for tablet e716270ffcf44039b075023fd0c2d2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.319495 893 tablet_service.cc:1467] Processing CreateTablet for tablet 33c7eef84a0c46de990aee95ada1e48d (DEFAULT_TABLE table=default.Table_e6a87107149f421e93c2ef7e26400ef1 [id=7878e437152b4d90b49864428cd94ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.324473 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.324698 895 tablet_service.cc:1514] Processing DeleteTablet for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.325016 885 tablet_service.cc:1467] Processing CreateTablet for tablet b8532291b4d0482690f3eed45c0f1f5b (DEFAULT_TABLE table=default.Table_604867c71c1740bf93813e9a6c0cb268 [id=29ed39cbeeb7486993274f97a291596b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.325410 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c7eef84a0c46de990aee95ada1e48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.326438 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8532291b4d0482690f3eed45c0f1f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.327565 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8532291b4d0482690f3eed45c0f1f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.333400 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 31de9638669040db9674d3aa7df32103 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.333647 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0bfb44367a4572a9507af0448af846 (DEFAULT_TABLE table=default.Table_1340076b6d8146429e90b485b3b58728 [id=141a963a55d14eba985268a2767410d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.333616 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_eb2e935fd7fb43c0ba59c503c575d803" } new_table_name: "default.Table_8c1d631caa444686bedde93287738a48" modify_external_catalogs: true
I20250109 20:52:52.330040 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.335497 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a0bfb44367a4572a9507af0448af846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.336126 1176 tablet_bootstrap.cc:492] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:52.336385 1176 ts_tablet_manager.cc:1397] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.114s user 0.008s sys 0.000s
I20250109 20:52:52.338640 1176 raft_consensus.cc:357] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.338929 1176 raft_consensus.cc:383] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.339107 1176 raft_consensus.cc:738] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.339411 1176 consensus_queue.cc:260] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.340137 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4c9509d9c9b04d1db58e1d8b871579d2" } new_table_name: "default.Table_2fdd49df493a4c26bcdea27254701757" modify_external_catalogs: true
I20250109 20:52:52.342008 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e6adec6051bc4812bb9fbc119614d604"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.342578 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6adec6051bc4812bb9fbc119614d604 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:52.342877 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 617029392e994f44a60c71ba2e340c57 consensus state has no leader
I20250109 20:52:52.345019 1176 ts_tablet_manager.cc:1428] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250109 20:52:52.348467 1176 tablet_bootstrap.cc:492] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:52.352133 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9151edfc060b4199b6ca8ec93a8ab836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.352595 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9151edfc060b4199b6ca8ec93a8ab836 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:52.359668 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4b31bd4072941d5aaf8c43dada3ccc7 consensus state has no leader
I20250109 20:52:52.360939 744 tablet_service.cc:1514] Processing DeleteTablet for tablet feb9b0f722eb44b0acd5cb3bbb32c527 with delete_type TABLET_DATA_DELETED (replaced by tablet d4b939cdea1c4719b9b3918aabbdfc0e) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.366080 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 221461us
I20250109 20:52:52.366261 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 31de9638669040db9674d3aa7df32103 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.370555 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet feb9b0f722eb44b0acd5cb3bbb32c527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet feb9b0f722eb44b0acd5cb3bbb32c527 already in progress: creating tablet
I20250109 20:52:52.380791 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b06e8616df461d89c58d414fc0cfbd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:52.383054 1053 raft_consensus.cc:3054] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.383075 739 tablet_service.cc:1467] Processing CreateTablet for tablet 33c7eef84a0c46de990aee95ada1e48d (DEFAULT_TABLE table=default.Table_e6a87107149f421e93c2ef7e26400ef1 [id=7878e437152b4d90b49864428cd94ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.384063 748 tablet_service.cc:1514] Processing DeleteTablet for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.384671 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 31de9638669040db9674d3aa7df32103 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31de9638669040db9674d3aa7df32103 already in progress: creating tablet
I20250109 20:52:52.384933 744 tablet_service.cc:1514] Processing DeleteTablet for tablet e716270ffcf44039b075023fd0c2d2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.385885 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0bfb44367a4572a9507af0448af846 (DEFAULT_TABLE table=default.Table_1340076b6d8146429e90b485b3b58728 [id=141a963a55d14eba985268a2767410d4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:52.386425 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbef1d27c5394fc1b6354c5d6f3a8f1f already in progress: creating tablet
W20250109 20:52:52.386679 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e716270ffcf44039b075023fd0c2d2a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e716270ffcf44039b075023fd0c2d2a3 already in progress: creating tablet
I20250109 20:52:52.387022 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67be9533983443cbb8fc52031216a973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.387804 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8532291b4d0482690f3eed45c0f1f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.391974 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 402280f75e51463a865c0fb521d0fd8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.393226 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a0bfb44367a4572a9507af0448af846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.393765 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 16c9f7274a7542e89a19bd4afcf89205 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.394093 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d2a69f7c330144e6bd09c7df8c65821e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.394635 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2a69f7c330144e6bd09c7df8c65821e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.394683 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_036a876d87ae4027b9002e151ed56074"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.395174 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_036a876d87ae4027b9002e151ed56074 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.399339 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c7eef84a0c46de990aee95ada1e48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.399691 1176 tablet_bootstrap.cc:654] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.404356 1053 raft_consensus.cc:2463] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:52.407960 1166 ts_tablet_manager.cc:1918] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.408167 1166 log.cc:1198] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/cba68160be6145ddbbd2019c9b5f0076
I20250109 20:52:52.408803 1166 ts_tablet_manager.cc:1939] T cba68160be6145ddbbd2019c9b5f0076 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:52.410036 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet cba68160be6145ddbbd2019c9b5f0076 (table default.Table_b70135c5a9b548538e791703e28af8f7 [id=f556fe2500bb47efba4056c36ca34823]) successfully deleted
W20250109 20:52:52.410311 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cd596614521a4a92a71b781b162f70b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd596614521a4a92a71b781b162f70b6 already in progress: creating tablet
W20250109 20:52:52.410678 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2e1f70532d3045499f2c3ef554740247 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e1f70532d3045499f2c3ef554740247 already in progress: creating tablet
W20250109 20:52:52.411240 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet da7f67026b164e0faad128d34188afb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da7f67026b164e0faad128d34188afb5 already in progress: creating tablet
W20250109 20:52:52.414831 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2e1f70532d3045499f2c3ef554740247 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e1f70532d3045499f2c3ef554740247 already in progress: creating tablet
I20250109 20:52:52.417424 1166 tablet_replica.cc:331] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:52.417708 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9d13168ca13c400a8a2d9608293ff8c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d13168ca13c400a8a2d9608293ff8c3 already in progress: creating tablet
I20250109 20:52:52.420995 1158 tablet_bootstrap.cc:492] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.421181 1158 ts_tablet_manager.cc:1397] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.315s user 0.008s sys 0.011s
I20250109 20:52:52.421217 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2ea95ef93df94b41a15c6f2ce7252567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.421705 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ea95ef93df94b41a15c6f2ce7252567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.422821 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_057129a1f1c74f8ca760ab2430165af0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.423787 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_057129a1f1c74f8ca760ab2430165af0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.424979 1166 raft_consensus.cc:2238] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.425180 1166 raft_consensus.cc:2267] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.427139 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 16c9f7274a7542e89a19bd4afcf89205
I20250109 20:52:52.427497 452 catalog_manager.cc:6596] ReplaceTablet: tablet 16c9f7274a7542e89a19bd4afcf89205 deleted and replaced by tablet 56fac585b2904ab490a6882dd5252a9a
I20250109 20:52:52.429032 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a34657904fd9418f8760104fbcc555e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:52.429256 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 with delete_type TABLET_DATA_DELETED (replaced by tablet 56fac585b2904ab490a6882dd5252a9a) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.429447 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a34657904fd9418f8760104fbcc555e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.430191 1158 raft_consensus.cc:357] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.430379 1158 raft_consensus.cc:383] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.431667 1158 raft_consensus.cc:738] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.431907 1158 consensus_queue.cc:260] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.432665 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 with delete_type TABLET_DATA_DELETED (replaced by tablet 56fac585b2904ab490a6882dd5252a9a) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.433135 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3b113c9e02ff40f2b63eab79960b09bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.433682 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b113c9e02ff40f2b63eab79960b09bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.434585 1158 ts_tablet_manager.cc:1428] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250109 20:52:52.440003 1158 tablet_bootstrap.cc:492] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:52.440320 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 16c9f7274a7542e89a19bd4afcf89205 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16c9f7274a7542e89a19bd4afcf89205 already in progress: creating tablet
I20250109 20:52:52.441757 1158 tablet_bootstrap.cc:654] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.452226 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f73f4c073c5a4d26b61421f75c8da4fe" } new_table_name: "default.Table_4f8710d177654a9c81c383e7aac0841b" modify_external_catalogs: true
I20250109 20:52:52.453437 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1cf2c16f6ff048719047b27308f7cf99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.453881 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cf2c16f6ff048719047b27308f7cf99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.475034 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5c4d17cdde7b4233916baefff36aeb42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.475673 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c4d17cdde7b4233916baefff36aeb42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.477571 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f54840d0459f412da09f1eb1bd75ae74" } new_table_name: "default.Table_a5679436e31841a59474006e2efcc11f" modify_external_catalogs: true
I20250109 20:52:52.479271 1166 ts_tablet_manager.cc:1905] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.486397 1160 ts_tablet_manager.cc:1918] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.491315 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1b4118488f5545d79852964d4b14bfe5" } new_table_name: "default.Table_b527bb4ba85142d1aabf4d172f9d8e36" modify_external_catalogs: true
I20250109 20:52:52.495560 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de7acfa7f941caa94bdbb888cf841e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.495946 1157 tablet_replica.cc:331] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:52.496177 1157 raft_consensus.cc:2238] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.496366 1157 raft_consensus.cc:2267] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.497040 1157 ts_tablet_manager.cc:1905] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.503037 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 72cc0bbbb5584f14ba4d6c62211a12c5 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.503451 1157 ts_tablet_manager.cc:1918] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.503690 1157 log.cc:1198] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/41de7acfa7f941caa94bdbb888cf841e
I20250109 20:52:52.504215 1157 ts_tablet_manager.cc:1939] T 41de7acfa7f941caa94bdbb888cf841e P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:52.505340 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 41de7acfa7f941caa94bdbb888cf841e (table default.Table_65cb0c2ed23b450ca192c6f61becbc51 [id=2504d7b1b64f477a8b7763d5d0a9ec6e]) successfully deleted
I20250109 20:52:52.505936 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 with delete_type TABLET_DATA_DELETED (replaced by tablet 56fac585b2904ab490a6882dd5252a9a) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.508044 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c4b9fe08755e4cb1b56978cbda1f2025" } modify_external_catalogs: true
W20250109 20:52:52.508447 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d2ad301c11943b69b59be18d6775513 consensus state has no leader
I20250109 20:52:52.508965 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f739863bb39f41698e7e5a083fe352dd" } modify_external_catalogs: true
I20250109 20:52:52.509182 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f739863bb39f41698e7e5a083fe352dd" } modify_external_catalogs: true
I20250109 20:52:52.508970 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6f5675a262804b5b98db035a4c98ddcf" } modify_external_catalogs: true
I20250109 20:52:52.510280 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6f5675a262804b5b98db035a4c98ddcf" } modify_external_catalogs: true
I20250109 20:52:52.508996 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c4b9fe08755e4cb1b56978cbda1f2025" } modify_external_catalogs: true
I20250109 20:52:52.517714 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7de47568081c4d11a33a67edd6898f8f" } modify_external_catalogs: true
I20250109 20:52:52.517897 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7de47568081c4d11a33a67edd6898f8f" } modify_external_catalogs: true
I20250109 20:52:52.522269 1158 tablet_bootstrap.cc:492] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.522490 1158 ts_tablet_manager.cc:1397] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.009s
I20250109 20:52:52.523378 1158 raft_consensus.cc:357] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.523653 1158 raft_consensus.cc:383] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.523789 1158 raft_consensus.cc:738] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.524142 1158 consensus_queue.cc:260] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.524699 1158 ts_tablet_manager.cc:1428] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:52.525069 1158 tablet_bootstrap.cc:492] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.526686 1158 tablet_bootstrap.cc:654] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.531134 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_79f5ff4b60444c1fb9e227b39c89e4f4" } modify_external_catalogs: true
I20250109 20:52:52.531339 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_79f5ff4b60444c1fb9e227b39c89e4f4" } modify_external_catalogs: true
I20250109 20:52:52.532053 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 85a95a31468b431fb4304cf6680016da from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.505060 1165 tablet_bootstrap.cc:492] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:52.532285 1165 ts_tablet_manager.cc:1397] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.272s user 0.012s sys 0.019s
I20250109 20:52:52.532975 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.533042 1165 raft_consensus.cc:357] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.533247 1157 tablet_replica.cc:331] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:52.533246 1165 raft_consensus.cc:383] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.533310 1165 raft_consensus.cc:738] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.533421 1157 raft_consensus.cc:2238] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.533572 1157 raft_consensus.cc:2267] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.533511 1165 consensus_queue.cc:260] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.533962 1165 ts_tablet_manager.cc:1428] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:52.534389 1165 tablet_bootstrap.cc:492] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:52.535986 1165 tablet_bootstrap.cc:654] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.538811 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c03524194a7b4e34a20424091e308ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.550793 1160 log.cc:1198] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2b9feb9de9dc4d22bafc0f7707943113
I20250109 20:52:52.551208 1160 ts_tablet_manager.cc:1939] T 2b9feb9de9dc4d22bafc0f7707943113 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:52.551708 1160 tablet_replica.cc:331] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:52.551880 1160 raft_consensus.cc:2238] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.552059 1160 raft_consensus.cc:2267] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.552570 1160 ts_tablet_manager.cc:1905] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.555569 1166 ts_tablet_manager.cc:1918] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.555737 1166 log.cc:1198] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b77c8d6adbf44375aff2782a83643255
I20250109 20:52:52.556054 1166 ts_tablet_manager.cc:1939] T b77c8d6adbf44375aff2782a83643255 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:52.556515 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2b9feb9de9dc4d22bafc0f7707943113 (table default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]) successfully deleted
I20250109 20:52:52.557042 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b77c8d6adbf44375aff2782a83643255 (table default.Table_a5bb2f6d26e3423e93672a1954c238a7 [id=43c66cf1f4554c79b35bd2689050a945]) successfully deleted
W20250109 20:52:52.558717 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c03310d8c1ff4b3a82dd0ce35992a319 has no consensus state
W20250109 20:52:52.564472 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 176791us
I20250109 20:52:52.564814 713 leader_election.cc:304] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:52.565299 1678 raft_consensus.cc:2798] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:52.565418 1678 raft_consensus.cc:695] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:52.565649 1678 consensus_queue.cc:237] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.566184 1166 tablet_replica.cc:331] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:52.566375 1166 raft_consensus.cc:2238] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.566516 1166 raft_consensus.cc:2267] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:52.567131 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet bd62a3ae64a143998bb1172b3a97c69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd62a3ae64a143998bb1172b3a97c69f already in progress: creating tablet
I20250109 20:52:52.566952 1166 ts_tablet_manager.cc:1905] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:52.567478 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f2383d9770834b1aae290308b4185299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2383d9770834b1aae290308b4185299 already in progress: creating tablet
I20250109 20:52:52.570412 1176 tablet_bootstrap.cc:492] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:52.570627 1176 ts_tablet_manager.cc:1397] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.222s user 0.005s sys 0.008s
I20250109 20:52:52.571460 1176 raft_consensus.cc:357] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.571722 1176 raft_consensus.cc:383] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.571799 1176 raft_consensus.cc:738] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.572013 1176 consensus_queue.cc:260] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.572470 1176 ts_tablet_manager.cc:1428] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:52.573078 1157 ts_tablet_manager.cc:1905] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.579716 1157 ts_tablet_manager.cc:1918] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.579879 1157 log.cc:1198] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/49c6f1833ff1453f90a6ec6435248a02
I20250109 20:52:52.580264 1157 ts_tablet_manager.cc:1939] T 49c6f1833ff1453f90a6ec6435248a02 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:52.581493 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 49c6f1833ff1453f90a6ec6435248a02 (table default.Table_d59d31f7bc514ff38a99d27cbe8170f9 [id=5514c15f7b8c406288bec6394be1d99b]) successfully deleted
I20250109 20:52:52.584652 1165 tablet_bootstrap.cc:492] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:52.584875 1165 ts_tablet_manager.cc:1397] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.008s
I20250109 20:52:52.585741 1165 raft_consensus.cc:357] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.585996 1165 raft_consensus.cc:383] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.586130 1165 raft_consensus.cc:738] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.586360 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8303e354c4f54a6fa05206c788701396" } new_table_name: "default.Table_5e53bf0c9fce4d04a077919ffb080698" modify_external_catalogs: true
I20250109 20:52:52.586405 1165 consensus_queue.cc:260] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.586890 1165 ts_tablet_manager.cc:1428] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:52.592939 1165 tablet_bootstrap.cc:492] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:52:52.594376 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f03d4e679a1c4793bd83cd979acced88 consensus state has no leader
I20250109 20:52:52.596616 1160 ts_tablet_manager.cc:1918] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.596755 1160 log.cc:1198] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f0e8ee4b86a04199865acd5f69cda384
I20250109 20:52:52.597148 1160 ts_tablet_manager.cc:1939] T f0e8ee4b86a04199865acd5f69cda384 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:52.598110 1165 tablet_bootstrap.cc:654] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.602862 1176 tablet_bootstrap.cc:492] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:52.609054 1176 tablet_bootstrap.cc:654] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.609555 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.610852 1158 tablet_bootstrap.cc:492] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.611039 1158 ts_tablet_manager.cc:1397] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.086s user 0.000s sys 0.013s
I20250109 20:52:52.611786 1158 raft_consensus.cc:357] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.612052 1158 raft_consensus.cc:383] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.612159 1158 raft_consensus.cc:738] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.612402 1158 consensus_queue.cc:260] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.612919 1158 ts_tablet_manager.cc:1428] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:52.614813 1160 tablet_replica.cc:331] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:52.615038 1160 raft_consensus.cc:2238] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.615202 1158 tablet_bootstrap.cc:492] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.615206 1160 raft_consensus.cc:2267] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.616007 1165 tablet_bootstrap.cc:492] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:52.616185 1165 ts_tablet_manager.cc:1397] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20250109 20:52:52.616969 1158 tablet_bootstrap.cc:654] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.616868 1165 raft_consensus.cc:357] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.617180 1165 raft_consensus.cc:383] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.617290 1165 raft_consensus.cc:738] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.617578 1165 consensus_queue.cc:260] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.618088 1165 ts_tablet_manager.cc:1428] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:52.620043 1165 tablet_bootstrap.cc:492] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:52.625469 1166 ts_tablet_manager.cc:1918] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.625612 1166 log.cc:1198] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6fb563bd6802473fafedd08bb10d15c1
I20250109 20:52:52.625931 1165 tablet_bootstrap.cc:654] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.625945 1166 ts_tablet_manager.cc:1939] T 6fb563bd6802473fafedd08bb10d15c1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:52.630084 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 49c6f1833ff1453f90a6ec6435248a02 because the tablet was not found. No further retry: Not found: Tablet not found: 49c6f1833ff1453f90a6ec6435248a02
I20250109 20:52:52.630349 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6fb563bd6802473fafedd08bb10d15c1 (table default.Table_d2ee28b720884b02a77177a6c829d85a [id=dccf417b39fc4d2d87812e340ecd842e]) successfully deleted
I20250109 20:52:52.631728 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a0f0901b5b4e4116ac67c1bcc180251a" } new_table_name: "default.Table_6c4a9feb3c4b4f86a11bb8ff982d4304" modify_external_catalogs: true
I20250109 20:52:52.631907 1158 tablet_bootstrap.cc:492] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.632086 1158 ts_tablet_manager.cc:1397] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20250109 20:52:52.630636 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f0e8ee4b86a04199865acd5f69cda384 (table default.Table_b683408175de45c0bf7d7f7e14e15447 [id=2a713bcff6ae4df0a2a9ad75a6986394]) successfully deleted
I20250109 20:52:52.633391 1158 raft_consensus.cc:357] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.633598 1158 raft_consensus.cc:383] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.633671 1158 raft_consensus.cc:738] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.633915 1158 consensus_queue.cc:260] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.634488 1158 ts_tablet_manager.cc:1428] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:52.634682 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c6f1833ff1453f90a6ec6435248a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.634785 1158 tablet_bootstrap.cc:492] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:52.635162 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet bd62a3ae64a143998bb1172b3a97c69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd62a3ae64a143998bb1172b3a97c69f already in progress: creating tablet
W20250109 20:52:52.635473 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f2383d9770834b1aae290308b4185299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2383d9770834b1aae290308b4185299 already in progress: creating tablet
W20250109 20:52:52.635756 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d9f18f244a344272bba2bc085686b61d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f18f244a344272bba2bc085686b61d already in progress: creating tablet
W20250109 20:52:52.635951 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2f5609eaf86d4e8ea3d3da41243c927f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f5609eaf86d4e8ea3d3da41243c927f already in progress: creating tablet
W20250109 20:52:52.636163 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet deedac68322e460b8d62adbe849de3f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deedac68322e460b8d62adbe849de3f2 already in progress: creating tablet
W20250109 20:52:52.636349 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 12e97ff530df46ae80eb1b5615a55a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e97ff530df46ae80eb1b5615a55a3f already in progress: creating tablet
W20250109 20:52:52.636555 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet feb9b0f722eb44b0acd5cb3bbb32c527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet feb9b0f722eb44b0acd5cb3bbb32c527 already in progress: creating tablet
W20250109 20:52:52.636777 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 37ee437300414eccba9926e824f72f27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ee437300414eccba9926e824f72f27 already in progress: creating tablet
W20250109 20:52:52.636993 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 31de9638669040db9674d3aa7df32103 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31de9638669040db9674d3aa7df32103 already in progress: creating tablet
W20250109 20:52:52.637181 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e716270ffcf44039b075023fd0c2d2a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e716270ffcf44039b075023fd0c2d2a3 already in progress: creating tablet
I20250109 20:52:52.638725 1176 tablet_bootstrap.cc:492] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:52.638864 1176 ts_tablet_manager.cc:1397] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250109 20:52:52.641364 1165 tablet_bootstrap.cc:492] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:52.641376 1176 raft_consensus.cc:357] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.641548 1176 raft_consensus.cc:383] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.641625 1176 raft_consensus.cc:738] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.641752 1165 ts_tablet_manager.cc:1397] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250109 20:52:52.641809 1176 consensus_queue.cc:260] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.642648 1165 raft_consensus.cc:357] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.642887 1165 raft_consensus.cc:383] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.642997 1165 raft_consensus.cc:738] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.645181 1165 consensus_queue.cc:260] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.646000 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cf33c8d126254dd580ef3fb97ecad07f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.646520 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf33c8d126254dd580ef3fb97ecad07f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.649693 1165 ts_tablet_manager.cc:1428] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250109 20:52:52.652312 1165 tablet_bootstrap.cc:492] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:52:52.654338 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f407b76780974cf7b3635fa1bf1585ec consensus state has no leader
I20250109 20:52:52.655838 1657 raft_consensus.cc:491] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.656002 1657 raft_consensus.cc:513] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.656591 1657 leader_election.cc:290] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:52.657475 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab7ca098694464faec002cec05558ac" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:52.657673 1053 raft_consensus.cc:2463] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:52.658017 854 leader_election.cc:304] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:52.658349 1158 tablet_bootstrap.cc:654] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.658767 1176 ts_tablet_manager.cc:1428] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250109 20:52:52.658841 1657 raft_consensus.cc:2798] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:52.658922 1657 raft_consensus.cc:491] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:52.659008 1657 raft_consensus.cc:3054] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.661521 1657 raft_consensus.cc:513] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.658593 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab7ca098694464faec002cec05558ac" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.662822 1657 leader_election.cc:290] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:52.663070 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab7ca098694464faec002cec05558ac" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:52.663264 1053 raft_consensus.cc:3054] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.643496 1160 ts_tablet_manager.cc:1905] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.665418 1165 tablet_bootstrap.cc:654] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.665884 1053 raft_consensus.cc:2463] T 2ab7ca098694464faec002cec05558ac P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:52.666215 854 leader_election.cc:304] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:52.666545 1657 raft_consensus.cc:2798] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election won for term 1
W20250109 20:52:52.667696 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a99e7e38d8a40268faa7241d72850d6 has no consensus state
I20250109 20:52:52.670204 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]). Expected version 1 got 0
I20250109 20:52:52.670428 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e5d7cc7fe84d481a974efb955fbeb6e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.670944 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5d7cc7fe84d481a974efb955fbeb6e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.671447 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8eca6041a4b143d5944be9db156dbb1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.671955 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8eca6041a4b143d5944be9db156dbb1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.673444 774 raft_consensus.cc:2463] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:52.673758 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.673992 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.674772 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab7ca098694464faec002cec05558ac" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:52.679337 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 9c7ed9e8c921491c9d911d662f500451 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.682261 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu pid 1296
I20250109 20:52:52.686118 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a82f87794b0d48f09f851336ce4e89d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.687117 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a82f87794b0d48f09f851336ce4e89d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.687261 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9c7ed9e8c921491c9d911d662f500451
I20250109 20:52:52.688244 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7ed9e8c921491c9d911d662f500451 with delete_type TABLET_DATA_DELETED (replaced by tablet 36bc408ae5174be89dbc2a13de464dd4) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.688895 446 catalog_manager.cc:6596] ReplaceTablet: tablet 9c7ed9e8c921491c9d911d662f500451 deleted and replaced by tablet 36bc408ae5174be89dbc2a13de464dd4
I20250109 20:52:52.689239 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cd252806276c4bc28ada21e4c72c6c08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.689832 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd252806276c4bc28ada21e4c72c6c08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.691138 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7ed9e8c921491c9d911d662f500451 with delete_type TABLET_DATA_DELETED (replaced by tablet 36bc408ae5174be89dbc2a13de464dd4) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.693092 1657 raft_consensus.cc:695] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Running, Role: LEADER
I20250109 20:52:52.693409 1657 consensus_queue.cc:237] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.694285 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5e1c8b41b7ee4ac08260bfdefdc0555f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.698100 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1c8b41b7ee4ac08260bfdefdc0555f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.699311 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6cba456285b14fd7921032996e6e4c89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.701510 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cba456285b14fd7921032996e6e4c89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.706501 774 raft_consensus.cc:3054] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.706703 1176 tablet_bootstrap.cc:492] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:52.707728 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7ed9e8c921491c9d911d662f500451 with delete_type TABLET_DATA_DELETED (replaced by tablet 36bc408ae5174be89dbc2a13de464dd4) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.707999 1157 tablet_replica.cc:331] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:52.708395 1157 raft_consensus.cc:2238] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:52.708348 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3219be93dffd45bb89a8ac2f801d567a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:52.708778 1157 raft_consensus.cc:2267] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:52.709065 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3219be93dffd45bb89a8ac2f801d567a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.710019 1176 tablet_bootstrap.cc:654] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.711097 1157 ts_tablet_manager.cc:1905] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.716552 1176 tablet_bootstrap.cc:492] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:52.716925 1176 ts_tablet_manager.cc:1397] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20250109 20:52:52.717921 774 raft_consensus.cc:2463] T 2ab7ca098694464faec002cec05558ac P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
W20250109 20:52:52.718412 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d9f18f244a344272bba2bc085686b61d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f18f244a344272bba2bc085686b61d already in progress: creating tablet
W20250109 20:52:52.719054 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2f5609eaf86d4e8ea3d3da41243c927f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f5609eaf86d4e8ea3d3da41243c927f already in progress: creating tablet
I20250109 20:52:52.719177 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d5fcf8d73704433193c50e16d0fe4184" } new_table_name: "default.Table_887acb86017941c4b8636c1008d835d9" modify_external_catalogs: true
W20250109 20:52:52.719686 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet deedac68322e460b8d62adbe849de3f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deedac68322e460b8d62adbe849de3f2 already in progress: creating tablet
W20250109 20:52:52.720647 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 12e97ff530df46ae80eb1b5615a55a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e97ff530df46ae80eb1b5615a55a3f already in progress: creating tablet
W20250109 20:52:52.721247 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet feb9b0f722eb44b0acd5cb3bbb32c527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet feb9b0f722eb44b0acd5cb3bbb32c527 already in progress: creating tablet
W20250109 20:52:52.721772 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 37ee437300414eccba9926e824f72f27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ee437300414eccba9926e824f72f27 already in progress: creating tablet
I20250109 20:52:52.721892 1157 ts_tablet_manager.cc:1918] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
W20250109 20:52:52.722352 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e716270ffcf44039b075023fd0c2d2a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e716270ffcf44039b075023fd0c2d2a3 already in progress: creating tablet
W20250109 20:52:52.723127 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 31de9638669040db9674d3aa7df32103 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31de9638669040db9674d3aa7df32103 already in progress: creating tablet
W20250109 20:52:52.724022 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbef1d27c5394fc1b6354c5d6f3a8f1f already in progress: creating tablet
I20250109 20:52:52.724228 1157 log.cc:1198] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/9c7ed9e8c921491c9d911d662f500451
I20250109 20:52:52.724367 1166 tablet_replica.cc:331] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:52.724665 1166 raft_consensus.cc:2238] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.724876 1157 ts_tablet_manager.cc:1939] T 9c7ed9e8c921491c9d911d662f500451 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:52.724951 1166 raft_consensus.cc:2267] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:52.726444 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59ebeffe516c4c239ae2ee9040276e53 consensus state has no leader
I20250109 20:52:52.726756 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_76d70836419947c78b06922a4fe21a2c [id=53718e55e4344456851f408909cc8170]) successfully deleted
I20250109 20:52:52.727746 1176 raft_consensus.cc:357] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.728015 1176 raft_consensus.cc:383] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.728093 1176 raft_consensus.cc:738] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.728322 1176 consensus_queue.cc:260] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.728914 1176 ts_tablet_manager.cc:1428] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250109 20:52:52.729363 1176 tablet_bootstrap.cc:492] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:52.731423 1166 ts_tablet_manager.cc:1905] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.739188 1176 tablet_bootstrap.cc:654] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.739683 1165 tablet_bootstrap.cc:492] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:52.739900 1165 ts_tablet_manager.cc:1397] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.004s
I20250109 20:52:52.740698 1165 raft_consensus.cc:357] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.741708 1165 raft_consensus.cc:383] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.741843 1165 raft_consensus.cc:738] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.742156 1165 consensus_queue.cc:260] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.742758 1165 ts_tablet_manager.cc:1428] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:52.743184 1165 tablet_bootstrap.cc:492] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:52.745493 1165 tablet_bootstrap.cc:654] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.746338 1746 raft_consensus.cc:491] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.746485 1746 raft_consensus.cc:513] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.747246 1746 leader_election.cc:290] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:52.747711 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31b9785ee21418ba13b426ba94cfa9b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.748245 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31b9785ee21418ba13b426ba94cfa9b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:52.748311 990 leader_election.cc:343] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:52.748864 989 leader_election.cc:343] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.748993 989 leader_election.cc:304] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:52.749537 1658 raft_consensus.cc:2743] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:52.752208 1166 ts_tablet_manager.cc:1918] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.752436 1166 log.cc:1198] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/402280f75e51463a865c0fb521d0fd8a
I20250109 20:52:52.753057 1166 ts_tablet_manager.cc:1939] T 402280f75e51463a865c0fb521d0fd8a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:52.754221 1166 tablet_replica.cc:331] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:52.754312 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 402280f75e51463a865c0fb521d0fd8a (table default.Table_e740022397a8451e830a159fa021e461 [id=24e621d6ed7e4a78a53808f4a8c2dd23]) successfully deleted
I20250109 20:52:52.754575 1166 raft_consensus.cc:2238] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:52.754585 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 16c9f7274a7542e89a19bd4afcf89205 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16c9f7274a7542e89a19bd4afcf89205 already in progress: creating tablet
I20250109 20:52:52.754766 1166 raft_consensus.cc:2267] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.755568 1166 ts_tablet_manager.cc:1905] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.757367 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ff034ea8af80440fbe26a4c05c434f2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.765524 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff034ea8af80440fbe26a4c05c434f2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.767936 1158 tablet_bootstrap.cc:492] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.768189 1158 ts_tablet_manager.cc:1397] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.133s user 0.003s sys 0.009s
I20250109 20:52:52.769325 1158 raft_consensus.cc:357] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.769723 1158 raft_consensus.cc:383] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.769888 1158 raft_consensus.cc:738] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.770206 1158 consensus_queue.cc:260] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.770848 1158 ts_tablet_manager.cc:1428] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:52.771724 1166 ts_tablet_manager.cc:1918] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.774257 1166 log.cc:1198] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/fbde2eecdaf4470dbc74b0c59f31643c
I20250109 20:52:52.774696 1166 ts_tablet_manager.cc:1939] T fbde2eecdaf4470dbc74b0c59f31643c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:52.776048 1166 tablet_replica.cc:331] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:52.776504 1166 raft_consensus.cc:2238] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.776979 1158 tablet_bootstrap.cc:492] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.777182 1166 raft_consensus.cc:2267] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.777992 1166 ts_tablet_manager.cc:1905] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.778661 1158 tablet_bootstrap.cc:654] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.775830 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet fbde2eecdaf4470dbc74b0c59f31643c (table default.Table_56c8a1eb8409437c856c4af9ece36c61 [id=fd0bcd24e74342f299f6b060b684af79]) successfully deleted
I20250109 20:52:52.782997 1160 ts_tablet_manager.cc:1918] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.783136 1160 log.cc:1198] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/6acca65ecee94f8cbe3f2b89e744aa6c
I20250109 20:52:52.781212 1176 tablet_bootstrap.cc:492] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:52.783358 1176 ts_tablet_manager.cc:1397] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.002s
I20250109 20:52:52.783456 1160 ts_tablet_manager.cc:1939] T 6acca65ecee94f8cbe3f2b89e744aa6c P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:52.784197 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 6acca65ecee94f8cbe3f2b89e744aa6c (table default.Table_4792b8689a12402f9946e87ad47dcbbb [id=ec2d9d93307747a7b752255f2008500e]) successfully deleted
I20250109 20:52:52.784214 1176 raft_consensus.cc:357] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.784572 1176 raft_consensus.cc:383] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.784780 1176 raft_consensus.cc:738] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.785087 1176 consensus_queue.cc:260] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.785637 1176 ts_tablet_manager.cc:1428] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:52.782217 1746 raft_consensus.cc:491] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.787113 1746 raft_consensus.cc:513] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.787655 1746 leader_election.cc:290] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:52.788161 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b48900f57f47128e31b27275b1489c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.788595 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b48900f57f47128e31b27275b1489c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:52.788707 990 leader_election.cc:343] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:52.789570 989 leader_election.cc:343] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:52.789674 989 leader_election.cc:304] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:52.790089 1176 tablet_bootstrap.cc:492] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:52.793046 1746 raft_consensus.cc:2743] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:52.795913 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_603546c0274d4fb49ab48d08448c658d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.799129 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_603546c0274d4fb49ab48d08448c658d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.799561 1176 tablet_bootstrap.cc:654] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.803848 1166 ts_tablet_manager.cc:1918] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:52.806282 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbef1d27c5394fc1b6354c5d6f3a8f1f already in progress: creating tablet
I20250109 20:52:52.807862 1158 tablet_bootstrap.cc:492] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.808036 1158 ts_tablet_manager.cc:1397] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20250109 20:52:52.808682 1166 log.cc:1198] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ecb55a61d2e94cbb88d2f44fff1aa8d9
I20250109 20:52:52.809309 1166 ts_tablet_manager.cc:1939] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:52.809278 1158 raft_consensus.cc:357] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.809537 1158 raft_consensus.cc:383] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.809612 1158 raft_consensus.cc:738] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.809870 1158 consensus_queue.cc:260] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.810328 1158 ts_tablet_manager.cc:1428] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250109 20:52:52.810909 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 16c9f7274a7542e89a19bd4afcf89205 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16c9f7274a7542e89a19bd4afcf89205 already in progress: creating tablet
I20250109 20:52:52.811280 1158 tablet_bootstrap.cc:492] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.811947 1176 tablet_bootstrap.cc:492] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:52.812125 1176 ts_tablet_manager.cc:1397] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.003s
I20250109 20:52:52.812829 1176 raft_consensus.cc:357] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.816097 1165 tablet_bootstrap.cc:492] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:52.818631 1160 tablet_replica.cc:331] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:52.818631 1165 ts_tablet_manager.cc:1397] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.074s user 0.013s sys 0.016s
I20250109 20:52:52.818867 1160 raft_consensus.cc:2238] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.819093 1160 raft_consensus.cc:2267] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.820030 1165 raft_consensus.cc:357] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.820959 1165 raft_consensus.cc:383] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.821252 1165 raft_consensus.cc:738] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.821861 1165 consensus_queue.cc:260] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.823259 1657 raft_consensus.cc:491] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.823314 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_cbde6546bcd3407ebd6470f02d881bdd [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 4 got 0
I20250109 20:52:52.823406 1657 raft_consensus.cc:513] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.824254 1657 leader_election.cc:290] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:52.825096 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e90cd406e394166b3e8dedebf7fd0fc" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:52.825343 1053 raft_consensus.cc:2463] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:52.825402 1160 ts_tablet_manager.cc:1905] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.825940 854 leader_election.cc:304] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:52.826546 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e90cd406e394166b3e8dedebf7fd0fc" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.826756 774 raft_consensus.cc:2463] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:52.827374 1657 raft_consensus.cc:2798] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:52.827487 1657 raft_consensus.cc:491] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:52.827593 1657 raft_consensus.cc:3054] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.828527 1158 tablet_bootstrap.cc:654] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.830233 1657 raft_consensus.cc:513] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.830199 447 catalog_manager.cc:5526] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:52.830696 447 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:52.831468 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e90cd406e394166b3e8dedebf7fd0fc" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:52.831629 774 raft_consensus.cc:3054] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.832561 751 tablet_service.cc:1467] Processing CreateTablet for tablet c785c74ecee84e6fb5d27560cc7c6f28 (DEFAULT_TABLE table=default.Table_38191599779143059082c1ed7b3ccb76 [id=435e487c75e440b7a33280090ba8000c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.833032 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c785c74ecee84e6fb5d27560cc7c6f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.834561 774 raft_consensus.cc:2463] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:52.835124 854 leader_election.cc:304] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2; no voters:
I20250109 20:52:52.835422 1753 raft_consensus.cc:2798] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:52.836352 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e90cd406e394166b3e8dedebf7fd0fc" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:52.836578 1053 raft_consensus.cc:3054] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.839085 1053 raft_consensus.cc:2463] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:52.839720 1657 leader_election.cc:290] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:52.841663 897 tablet_service.cc:1467] Processing CreateTablet for tablet c785c74ecee84e6fb5d27560cc7c6f28 (DEFAULT_TABLE table=default.Table_38191599779143059082c1ed7b3ccb76 [id=435e487c75e440b7a33280090ba8000c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.842089 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c785c74ecee84e6fb5d27560cc7c6f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.847602 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_057129a1f1c74f8ca760ab2430165af0" } new_table_name: "default.Table_ab28e2a04600484e858415ada1017c00" modify_external_catalogs: true
I20250109 20:52:52.852061 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 04e46f4312844b6eb29d6718f4800f98
I20250109 20:52:52.853508 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e46f4312844b6eb29d6718f4800f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.853886 1176 raft_consensus.cc:383] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.854022 1176 raft_consensus.cc:738] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.854394 1176 consensus_queue.cc:260] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.855080 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 80da78917f0b4931ab1705371c1fe5dd from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:52.855674 1176 ts_tablet_manager.cc:1428] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.043s user 0.001s sys 0.000s
I20250109 20:52:52.856207 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet e64207928ef141118e7fc8fe7bb2379b with delete_type TABLET_DATA_DELETED (replaced by tablet 2797e1df3d12476f89fb5155523c3935) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.856433 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e46f4312844b6eb29d6718f4800f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.860692 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e46f4312844b6eb29d6718f4800f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.861996 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 04e46f4312844b6eb29d6718f4800f98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e46f4312844b6eb29d6718f4800f98 already in progress: creating tablet
I20250109 20:52:52.863411 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 189cd8079a9d4ca6b1703f4361d5ec5d
I20250109 20:52:52.863602 1176 tablet_bootstrap.cc:492] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:52.864248 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2ca83a49523b45be9ac7556b40746a83
I20250109 20:52:52.864342 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 189cd8079a9d4ca6b1703f4361d5ec5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.865123 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca83a49523b45be9ac7556b40746a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.865377 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 687db3e734af47339c2546970a71a539
I20250109 20:52:52.866953 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca83a49523b45be9ac7556b40746a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.867048 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 687db3e734af47339c2546970a71a539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.867089 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 189cd8079a9d4ca6b1703f4361d5ec5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.867740 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca83a49523b45be9ac7556b40746a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.867863 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 687db3e734af47339c2546970a71a539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.867954 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 189cd8079a9d4ca6b1703f4361d5ec5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.868743 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2ca83a49523b45be9ac7556b40746a83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ca83a49523b45be9ac7556b40746a83 already in progress: creating tablet
W20250109 20:52:52.868942 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 687db3e734af47339c2546970a71a539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 687db3e734af47339c2546970a71a539 already in progress: creating tablet
I20250109 20:52:52.869108 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 687db3e734af47339c2546970a71a539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:52.869139 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 189cd8079a9d4ca6b1703f4361d5ec5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 189cd8079a9d4ca6b1703f4361d5ec5d already in progress: creating tablet
I20250109 20:52:52.870533 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 (table default.Table_58060dd58dce4f7a8afa4d8c891c3ee3 [id=9a08f00bc4e74fc5861932c6f7cbbc58]) successfully deleted
I20250109 20:52:52.872217 1176 tablet_bootstrap.cc:654] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.875731 751 tablet_service.cc:1467] Processing CreateTablet for tablet 01174f43d56c4c96a3b754828fc30640 (DEFAULT_TABLE table=default.Table_f2e044c26f7c4b9d985304d701432239 [id=ac9488e51dbf45de8c2b8ba3de3991a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.876248 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01174f43d56c4c96a3b754828fc30640. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.872794 1166 tablet_replica.cc:331] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:52.874454 1030 tablet_service.cc:1467] Processing CreateTablet for tablet c785c74ecee84e6fb5d27560cc7c6f28 (DEFAULT_TABLE table=default.Table_38191599779143059082c1ed7b3ccb76 [id=435e487c75e440b7a33280090ba8000c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.877067 1166 raft_consensus.cc:2238] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.877297 1166 raft_consensus.cc:2267] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.877534 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c785c74ecee84e6fb5d27560cc7c6f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.879384 1166 ts_tablet_manager.cc:1905] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:52.885885 897 tablet_service.cc:1467] Processing CreateTablet for tablet 01174f43d56c4c96a3b754828fc30640 (DEFAULT_TABLE table=default.Table_f2e044c26f7c4b9d985304d701432239 [id=ac9488e51dbf45de8c2b8ba3de3991a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.886497 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01174f43d56c4c96a3b754828fc30640. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.895318 1158 tablet_bootstrap.cc:492] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.895470 1158 ts_tablet_manager.cc:1397] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.002s
I20250109 20:52:52.895799 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 72cc0bbbb5584f14ba4d6c62211a12c5
I20250109 20:52:52.896167 1158 raft_consensus.cc:357] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.896296 451 catalog_manager.cc:6596] ReplaceTablet: tablet 72cc0bbbb5584f14ba4d6c62211a12c5 deleted and replaced by tablet 76bd758fa2fa4cebbdf7faf3856f1941
I20250109 20:52:52.896351 1158 raft_consensus.cc:383] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.896422 1158 raft_consensus.cc:738] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.896616 1158 consensus_queue.cc:260] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.897040 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a1ed00f7954143c9ba9bf65d8a8a68d2
I20250109 20:52:52.897099 1158 ts_tablet_manager.cc:1428] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:52.897341 1158 tablet_bootstrap.cc:492] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.898895 1158 tablet_bootstrap.cc:654] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.898999 880 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.899220 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd758fa2fa4cebbdf7faf3856f1941) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:52.900202 1176 tablet_bootstrap.cc:492] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:52.900257 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.900336 1176 ts_tablet_manager.cc:1397] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20250109 20:52:52.900843 1753 raft_consensus.cc:695] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Running, Role: LEADER
I20250109 20:52:52.900868 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd758fa2fa4cebbdf7faf3856f1941) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.901005 739 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.901322 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd758fa2fa4cebbdf7faf3856f1941) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.901245 1753 consensus_queue.cc:237] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.902182 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8eca6041a4b143d5944be9db156dbb1e" } new_table_name: "default.Table_96de890f59cf48cb8d0b23bb9d8063fe" modify_external_catalogs: true
I20250109 20:52:52.903211 1176 raft_consensus.cc:357] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.903359 1176 raft_consensus.cc:383] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.903424 1176 raft_consensus.cc:738] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.903580 1176 consensus_queue.cc:260] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:52.903740 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0477b4a373d5435fac417a7c26c5103c" } new_table_name: "default.Table_88ff1094b93e4aedb7f622f1d97becaf" modify_external_catalogs: true
I20250109 20:52:52.905915 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 01174f43d56c4c96a3b754828fc30640 (DEFAULT_TABLE table=default.Table_f2e044c26f7c4b9d985304d701432239 [id=ac9488e51dbf45de8c2b8ba3de3991a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.906416 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b1b97f9ea66f4c7c89289094e6d57533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.906930 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1b97f9ea66f4c7c89289094e6d57533 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.906452 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:52.907657 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01174f43d56c4c96a3b754828fc30640. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.908056 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_95809365f485432a9035b1a4a42d6dae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.908507 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95809365f485432a9035b1a4a42d6dae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.910087 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b19d3251bcdf41eab27c97e89366d1c7" } new_table_name: "default.Table_487b0b221267468ba9603345f5030c8b" modify_external_catalogs: true
I20250109 20:52:52.910640 1766 raft_consensus.cc:491] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.910794 1766 raft_consensus.cc:513] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:52.911595 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72cc0bbbb5584f14ba4d6c62211a12c5 already in progress: creating tablet
W20250109 20:52:52.911872 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ed00f7954143c9ba9bf65d8a8a68d2 already in progress: creating tablet
I20250109 20:52:52.912737 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00aaf81b34445abb7502700f01d4f3f" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:52.913373 774 raft_consensus.cc:2463] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 0.
I20250109 20:52:52.913807 854 leader_election.cc:304] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2; no voters:
I20250109 20:52:52.914214 1657 raft_consensus.cc:2798] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:52.914345 1657 raft_consensus.cc:491] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:52.914479 1657 raft_consensus.cc:3054] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.916942 1657 raft_consensus.cc:513] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.918148 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00aaf81b34445abb7502700f01d4f3f" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:52.918401 774 raft_consensus.cc:3054] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.918614 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00aaf81b34445abb7502700f01d4f3f" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:52.918816 1053 raft_consensus.cc:3054] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:52.918774 739 tablet_service.cc:1467] Processing CreateTablet for tablet c368e39a840a4eceaf17180255fda3e9 (DEFAULT_TABLE table=default.Table_c57d7e4c6798461280d6ff32c28b05f7 [id=194cc65ab9e549a7b88cd577f5495e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.919186 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368e39a840a4eceaf17180255fda3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.921331 774 raft_consensus.cc:2463] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:52.921747 854 leader_election.cc:304] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2; no voters:
I20250109 20:52:52.922413 1053 raft_consensus.cc:2463] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1.
I20250109 20:52:52.925115 1771 raft_consensus.cc:2798] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:52.926609 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00aaf81b34445abb7502700f01d4f3f" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:52.926832 1053 raft_consensus.cc:2371] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 44d473955919435cb679edd5f8afe2c2 in term 1. Re-sending same reply.
I20250109 20:52:52.926914 1771 raft_consensus.cc:695] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Running, Role: LEADER
I20250109 20:52:52.927347 1766 leader_election.cc:290] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:52.927237 1771 consensus_queue.cc:237] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.925187 1657 leader_election.cc:290] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:52.932432 739 tablet_service.cc:1467] Processing CreateTablet for tablet d7e19d2421e34bd98e4401879bd3277d (DEFAULT_TABLE table=default.Table_6464281d788c4197b1e88847c593282c [id=3435cf574dd04055be1697f0fa80ccf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.932852 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e19d2421e34bd98e4401879bd3277d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.934940 1753 raft_consensus.cc:491] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.935159 1753 raft_consensus.cc:513] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.936406 1753 leader_election.cc:290] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:52.937012 854 leader_election.cc:343] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 49c6f1833ff1453f90a6ec6435248a02
W20250109 20:52:52.937572 854 leader_election.cc:343] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 49c6f1833ff1453f90a6ec6435248a02
I20250109 20:52:52.937695 854 leader_election.cc:304] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:52.938119 1753 raft_consensus.cc:2743] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:52.939535 1160 ts_tablet_manager.cc:1918] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:52.906807 1176 ts_tablet_manager.cc:1428] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20250109 20:52:52.940981 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08209ea49c894209abca6c0881f534f5 has no consensus state
I20250109 20:52:52.941831 446 catalog_manager.cc:5526] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d473955919435cb679edd5f8afe2c2 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "44d473955919435cb679edd5f8afe2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:52.942390 446 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:52.942415 1033 tablet_service.cc:1467] Processing CreateTablet for tablet d7e19d2421e34bd98e4401879bd3277d (DEFAULT_TABLE table=default.Table_6464281d788c4197b1e88847c593282c [id=3435cf574dd04055be1697f0fa80ccf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.942612 1176 tablet_bootstrap.cc:492] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:52.943413 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e19d2421e34bd98e4401879bd3277d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.943904 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 80da78917f0b4931ab1705371c1fe5dd
I20250109 20:52:52.944406 454 catalog_manager.cc:6596] ReplaceTablet: tablet 80da78917f0b4931ab1705371c1fe5dd deleted and replaced by tablet 3974c4196d4043be9ad6e0452ed9cb7b
I20250109 20:52:52.945636 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a34821f5c1e44d4cb26f0daa699282d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:52.945576 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c918439b959a4a4fa4f2c75750ff33ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.946110 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c918439b959a4a4fa4f2c75750ff33ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.946770 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da78917f0b4931ab1705371c1fe5dd with delete_type TABLET_DATA_DELETED (replaced by tablet 3974c4196d4043be9ad6e0452ed9cb7b) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:52.947418 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a34821f5c1e44d4cb26f0daa699282d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.948069 1678 raft_consensus.cc:491] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:52.948262 1678 raft_consensus.cc:513] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.949113 1678 leader_election.cc:290] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:52.950958 713 leader_election.cc:343] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 3000b819a4cc4960909b98febd400631
I20250109 20:52:52.951983 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da78917f0b4931ab1705371c1fe5dd with delete_type TABLET_DATA_DELETED (replaced by tablet 3974c4196d4043be9ad6e0452ed9cb7b) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:52.952802 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 80da78917f0b4931ab1705371c1fe5dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80da78917f0b4931ab1705371c1fe5dd already in progress: creating tablet
I20250109 20:52:52.953138 1176 tablet_bootstrap.cc:654] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.956269 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f5bea97ad34a41b1a53d274d1c918082"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.956936 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5bea97ad34a41b1a53d274d1c918082 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.957268 758 tablet_service.cc:1467] Processing CreateTablet for tablet b5bcf393703c4c9694190923216585d9 (DEFAULT_TABLE table=default.Table_46839e2fe6664d1b9d90e1677fa0d012 [id=3342ade149a84e67955fe136e3cfc683]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:52.957682 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcf393703c4c9694190923216585d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.965613 1166 ts_tablet_manager.cc:1918] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250109 20:52:52.965889 1166 log.cc:1198] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/9c7ed9e8c921491c9d911d662f500451
I20250109 20:52:52.966394 1166 ts_tablet_manager.cc:1939] T 9c7ed9e8c921491c9d911d662f500451 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:52.967618 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_76d70836419947c78b06922a4fe21a2c [id=53718e55e4344456851f408909cc8170]) successfully deleted
W20250109 20:52:52.968333 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 04e46f4312844b6eb29d6718f4800f98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e46f4312844b6eb29d6718f4800f98 already in progress: creating tablet
I20250109 20:52:52.968731 1166 tablet_replica.cc:331] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:52.969208 1030 tablet_service.cc:1467] Processing CreateTablet for tablet c368e39a840a4eceaf17180255fda3e9 (DEFAULT_TABLE table=default.Table_c57d7e4c6798461280d6ff32c28b05f7 [id=194cc65ab9e549a7b88cd577f5495e7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:52.970090 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c79ecf9db554b50b0a61325275b9bee consensus state has no leader
W20250109 20:52:52.970994 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1132c08b23ea4849a78b4809cbe44360 has no consensus state
I20250109 20:52:52.974669 1521 generic_service.cc:196] Checking for leaks (request via RPC)
I20250109 20:52:52.977217 1158 tablet_bootstrap.cc:492] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.977890 1158 ts_tablet_manager.cc:1397] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.001s
I20250109 20:52:52.978628 1158 raft_consensus.cc:357] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.978819 1158 raft_consensus.cc:383] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.978884 1158 raft_consensus.cc:738] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.979075 1158 consensus_queue.cc:260] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.980485 1158 ts_tablet_manager.cc:1428] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:52:52.980793 1158 tablet_bootstrap.cc:492] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.984249 1158 tablet_bootstrap.cc:654] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.992640 1158 tablet_bootstrap.cc:492] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:52.992816 1158 ts_tablet_manager.cc:1397] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20250109 20:52:52.994726 1158 raft_consensus.cc:357] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.994911 1158 raft_consensus.cc:383] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:52.995003 1158 raft_consensus.cc:738] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:52.996477 1158 consensus_queue.cc:260] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:52.997284 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_63fb1754d95144a3877b430bab3337b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:52.997925 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63fb1754d95144a3877b430bab3337b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:52.998214 1158 ts_tablet_manager.cc:1428] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:52.998430 1158 tablet_bootstrap.cc:492] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:52.999428 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_09933fbabe6e4a0885162da5f6e038db" } new_table_name: "default.Table_0c702c1df34a466da3d2c24f0150e438" modify_external_catalogs: true
I20250109 20:52:52.999795 1158 tablet_bootstrap.cc:654] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.002267 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:53.003733 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6c6662f9bad8425db77969d4150d9c38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.004247 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c6662f9bad8425db77969d4150d9c38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.009626 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.009938 1157 tablet_replica.cc:331] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:53.010051 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfd19df44fb463aab5812702cbbbfe4 (DEFAULT_TABLE table=default.Table_06aabfaaf12e49bfa1196ae2f5850d21 [id=7dc12c28159a4f56b8d7dadc1aefd296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.010180 1157 raft_consensus.cc:2238] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.010361 1157 raft_consensus.cc:2267] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.010458 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dfd19df44fb463aab5812702cbbbfe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.011366 1157 ts_tablet_manager.cc:1905] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.012985 756 tablet_service.cc:1467] Processing CreateTablet for tablet 78800ff1dd964d8ba8862e3f4af69336 (DEFAULT_TABLE table=default.Table_aab45704dc304088b76a3c55953aaef4 [id=4e9c4704ba694110994b6d961f8e25cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.013302 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78800ff1dd964d8ba8862e3f4af69336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.015769 452 catalog_manager.cc:5526] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d473955919435cb679edd5f8afe2c2 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "44d473955919435cb679edd5f8afe2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:53.016263 452 catalog_manager.cc:5526] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d473955919435cb679edd5f8afe2c2 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "44d473955919435cb679edd5f8afe2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:53.016564 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:53.017048 1158 tablet_bootstrap.cc:492] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.017199 1158 ts_tablet_manager.cc:1397] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20250109 20:52:53.017926 1158 raft_consensus.cc:357] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.019093 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_028d1b883d75407dbfa05dc1c7ba26cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.019667 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_028d1b883d75407dbfa05dc1c7ba26cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.020869 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3000b819a4cc4960909b98febd400631 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c65489a602141239c99e8e8dae22ab4) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.023072 758 tablet_service.cc:1514] Processing DeleteTablet for tablet c03524194a7b4e34a20424091e308ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.024574 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368e39a840a4eceaf17180255fda3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.024741 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c21468e8f0b14e899d210feae2ef211e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.025234 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c21468e8f0b14e899d210feae2ef211e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.025408 1157 ts_tablet_manager.cc:1918] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.025507 1157 log.cc:1198] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2a89a0b47d10414d9b6a9123af1f4ab3
I20250109 20:52:53.025727 1158 raft_consensus.cc:383] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.025825 1157 ts_tablet_manager.cc:1939] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:53.025859 1158 raft_consensus.cc:738] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.026093 1158 consensus_queue.cc:260] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.026520 1158 ts_tablet_manager.cc:1428] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:52:53.026618 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
I20250109 20:52:53.027351 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:53.028591 1157 tablet_replica.cc:331] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:52.969219 1166 raft_consensus.cc:2238] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:52.973177 1176 tablet_bootstrap.cc:492] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.007059 1023 tablet_service.cc:1467] Processing CreateTablet for tablet b5bcf393703c4c9694190923216585d9 (DEFAULT_TABLE table=default.Table_46839e2fe6664d1b9d90e1677fa0d012 [id=3342ade149a84e67955fe136e3cfc683]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.028867 1157 raft_consensus.cc:2238] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.028913 1176 ts_tablet_manager.cc:1397] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20250109 20:52:53.029037 1157 raft_consensus.cc:2267] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.029084 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcf393703c4c9694190923216585d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.029366 1158 tablet_bootstrap.cc:492] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.029803 1176 raft_consensus.cc:357] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.029994 1176 raft_consensus.cc:383] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.030061 1176 raft_consensus.cc:738] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.030270 1176 consensus_queue.cc:260] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.030740 1176 ts_tablet_manager.cc:1428] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:53.031005 1176 tablet_bootstrap.cc:492] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.032558 1176 tablet_bootstrap.cc:654] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.007745 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfd19df44fb463aab5812702cbbbfe4 (DEFAULT_TABLE table=default.Table_06aabfaaf12e49bfa1196ae2f5850d21 [id=7dc12c28159a4f56b8d7dadc1aefd296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.033456 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dfd19df44fb463aab5812702cbbbfe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.033632 1158 tablet_bootstrap.cc:654] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.033950 1157 ts_tablet_manager.cc:1905] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.041682 1166 raft_consensus.cc:2267] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:53.041899 1746 raft_consensus.cc:668] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I20250109 20:52:53.045114 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2a89a0b47d10414d9b6a9123af1f4ab3 (table default.Table_ef823ba5464d429db5aa6b8c3ccc1934 [id=70a6ff8c791e41038b082b9004740857]) successfully deleted
I20250109 20:52:53.053500 1166 ts_tablet_manager.cc:1905] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.056788 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 with delete_type TABLET_DATA_DELETED (replaced by tablet ed5ea5d588834f1b8b2b057ec324aadc) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:53.061533 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a2efcc2a6ec4b109845d8087491ca9f consensus state has no leader
I20250109 20:52:53.063593 1157 ts_tablet_manager.cc:1918] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.063767 1157 log.cc:1198] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/3000b819a4cc4960909b98febd400631
I20250109 20:52:53.064203 1157 ts_tablet_manager.cc:1939] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:52.940497 893 tablet_service.cc:1467] Processing CreateTablet for tablet d7e19d2421e34bd98e4401879bd3277d (DEFAULT_TABLE table=default.Table_6464281d788c4197b1e88847c593282c [id=3435cf574dd04055be1697f0fa80ccf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.065096 1160 log.cc:1198] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/c03524194a7b4e34a20424091e308ac4
I20250109 20:52:53.065423 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e19d2421e34bd98e4401879bd3277d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.065586 1160 ts_tablet_manager.cc:1939] T c03524194a7b4e34a20424091e308ac4 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:53.066349 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:52.945317 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da78917f0b4931ab1705371c1fe5dd with delete_type TABLET_DATA_DELETED (replaced by tablet 3974c4196d4043be9ad6e0452ed9cb7b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:53.069058 1160 tablet_replica.cc:331] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:52.949764 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3000b819a4cc4960909b98febd400631" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.069334 1160 raft_consensus.cc:2238] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.069530 1160 raft_consensus.cc:2267] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:52.953238 1165 ts_tablet_manager.cc:1428] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.134s user 0.000s sys 0.002s
I20250109 20:52:53.070245 1165 tablet_bootstrap.cc:492] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:53.070849 1160 ts_tablet_manager.cc:1905] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.072456 1165 tablet_bootstrap.cc:654] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:52.968719 885 tablet_service.cc:1467] Processing CreateTablet for tablet b5bcf393703c4c9694190923216585d9 (DEFAULT_TABLE table=default.Table_46839e2fe6664d1b9d90e1677fa0d012 [id=3342ade149a84e67955fe136e3cfc683]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.073343 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcf393703c4c9694190923216585d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.075073 1176 tablet_bootstrap.cc:492] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.075253 1176 ts_tablet_manager.cc:1397] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.000s
W20250109 20:52:53.076959 712 leader_election.cc:343] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.076828 1176 raft_consensus.cc:357] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.077105 1176 raft_consensus.cc:383] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.077102 712 leader_election.cc:304] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:53.077203 1176 raft_consensus.cc:738] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:52:53.077294 712 raft_consensus.cc:2681] T 3000b819a4cc4960909b98febd400631 P 09833ea46b7141d58c3899dace69e49f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:52:53.077459 1176 consensus_queue.cc:260] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.078444 1158 tablet_bootstrap.cc:492] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.078562 1158 ts_tablet_manager.cc:1397] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.006s
I20250109 20:52:53.079249 1158 raft_consensus.cc:357] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.079416 1158 raft_consensus.cc:383] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.079490 1158 raft_consensus.cc:738] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.079447 880 tablet_service.cc:1467] Processing CreateTablet for tablet a6630c808e354ecdb168332d5c3ea8da (DEFAULT_TABLE table=default.Table_7cbb3a94bca245ed8531800592512f15 [id=022381365ad94f0e9d7b75201260fd00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.079820 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6630c808e354ecdb168332d5c3ea8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.081234 1176 ts_tablet_manager.cc:1428] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250109 20:52:53.081569 1176 tablet_bootstrap.cc:492] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.082340 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet c03524194a7b4e34a20424091e308ac4 (table default.Table_df185d0c630644438a9e8cc5fe14446b [id=d89b416c6f8d4417a6ccb26b443b2ef4]) successfully deleted
I20250109 20:52:53.083563 1176 tablet_bootstrap.cc:654] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.083650 742 tablet_service.cc:1467] Processing CreateTablet for tablet 50afaf6c42a546b2ad7add93095a7556 (DEFAULT_TABLE table=default.Table_47985b842e6c4ab6adaf55ef480a90d2 [id=04f20922f02943c0bd5e77279a5a6c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.083606 1158 consensus_queue.cc:260] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.083992 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50afaf6c42a546b2ad7add93095a7556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.085323 1158 ts_tablet_manager.cc:1428] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250109 20:52:53.086313 1158 tablet_bootstrap.cc:492] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.087275 739 tablet_service.cc:1467] Processing CreateTablet for tablet a6630c808e354ecdb168332d5c3ea8da (DEFAULT_TABLE table=default.Table_7cbb3a94bca245ed8531800592512f15 [id=022381365ad94f0e9d7b75201260fd00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.089807 896 tablet_service.cc:1467] Processing CreateTablet for tablet 50afaf6c42a546b2ad7add93095a7556 (DEFAULT_TABLE table=default.Table_47985b842e6c4ab6adaf55ef480a90d2 [id=04f20922f02943c0bd5e77279a5a6c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.090237 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50afaf6c42a546b2ad7add93095a7556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.090833 1158 tablet_bootstrap.cc:654] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.093148 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 50afaf6c42a546b2ad7add93095a7556 (DEFAULT_TABLE table=default.Table_47985b842e6c4ab6adaf55ef480a90d2 [id=04f20922f02943c0bd5e77279a5a6c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.093773 758 tablet_service.cc:1467] Processing CreateTablet for tablet c03310d8c1ff4b3a82dd0ce35992a319 (DEFAULT_TABLE table=default.Table_a5679436e31841a59474006e2efcc11f [id=9ff2478f8378422e8c4155bc36705269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.095106 887 tablet_service.cc:1467] Processing CreateTablet for tablet c03310d8c1ff4b3a82dd0ce35992a319 (DEFAULT_TABLE table=default.Table_a5679436e31841a59474006e2efcc11f [id=9ff2478f8378422e8c4155bc36705269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.095661 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03310d8c1ff4b3a82dd0ce35992a319. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.096598 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c03310d8c1ff4b3a82dd0ce35992a319 (DEFAULT_TABLE table=default.Table_a5679436e31841a59474006e2efcc11f [id=9ff2478f8378422e8c4155bc36705269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.097831 756 tablet_service.cc:1467] Processing CreateTablet for tablet 60d3f97679c8408b9248b126a2218313 (DEFAULT_TABLE table=default.Table_30b1ffb8aacc48c6883ed04b539160e7 [id=eb5ba91d8d3a4f959e2b85bb77a154cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.098857 1027 tablet_service.cc:1467] Processing CreateTablet for tablet a6630c808e354ecdb168332d5c3ea8da (DEFAULT_TABLE table=default.Table_7cbb3a94bca245ed8531800592512f15 [id=022381365ad94f0e9d7b75201260fd00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.099602 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 60d3f97679c8408b9248b126a2218313 (DEFAULT_TABLE table=default.Table_30b1ffb8aacc48c6883ed04b539160e7 [id=eb5ba91d8d3a4f959e2b85bb77a154cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.100473 888 tablet_service.cc:1467] Processing CreateTablet for tablet 60d3f97679c8408b9248b126a2218313 (DEFAULT_TABLE table=default.Table_30b1ffb8aacc48c6883ed04b539160e7 [id=eb5ba91d8d3a4f959e2b85bb77a154cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.101058 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d3f97679c8408b9248b126a2218313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.102191 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 78800ff1dd964d8ba8862e3f4af69336 (DEFAULT_TABLE table=default.Table_aab45704dc304088b76a3c55953aaef4 [id=4e9c4704ba694110994b6d961f8e25cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.106092 1158 tablet_bootstrap.cc:492] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.106235 1158 ts_tablet_manager.cc:1397] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20250109 20:52:53.108397 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 3000b819a4cc4960909b98febd400631 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]) successfully deleted
I20250109 20:52:53.108664 1157 tablet_replica.cc:331] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:53.108825 1157 raft_consensus.cc:2238] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.108999 1157 raft_consensus.cc:2267] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.109573 1158 raft_consensus.cc:357] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.109768 1158 raft_consensus.cc:383] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.109833 1158 raft_consensus.cc:738] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.110029 1158 consensus_queue.cc:260] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.112025 1166 ts_tablet_manager.cc:1918] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.112226 1166 log.cc:1198] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/e64207928ef141118e7fc8fe7bb2379b
I20250109 20:52:53.112498 1158 ts_tablet_manager.cc:1428] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:52:53.112634 1166 ts_tablet_manager.cc:1939] T e64207928ef141118e7fc8fe7bb2379b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:53.112757 1158 tablet_bootstrap.cc:492] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.112919 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50afaf6c42a546b2ad7add93095a7556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.114228 1158 tablet_bootstrap.cc:654] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.114377 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03310d8c1ff4b3a82dd0ce35992a319. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.116868 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d3f97679c8408b9248b126a2218313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:52.983100 897 tablet_service.cc:1467] Processing CreateTablet for tablet c368e39a840a4eceaf17180255fda3e9 (DEFAULT_TABLE table=default.Table_c57d7e4c6798461280d6ff32c28b05f7 [id=194cc65ab9e549a7b88cd577f5495e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.006291 895 tablet_service.cc:1467] Processing CreateTablet for tablet 3dfd19df44fb463aab5812702cbbbfe4 (DEFAULT_TABLE table=default.Table_06aabfaaf12e49bfa1196ae2f5850d21 [id=7dc12c28159a4f56b8d7dadc1aefd296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.126098 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dfd19df44fb463aab5812702cbbbfe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.127354 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6630c808e354ecdb168332d5c3ea8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.007668 898 tablet_service.cc:1467] Processing CreateTablet for tablet 78800ff1dd964d8ba8862e3f4af69336 (DEFAULT_TABLE table=default.Table_aab45704dc304088b76a3c55953aaef4 [id=4e9c4704ba694110994b6d961f8e25cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.129091 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78800ff1dd964d8ba8862e3f4af69336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.133508 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78800ff1dd964d8ba8862e3f4af69336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.136579 1158 tablet_bootstrap.cc:492] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.136705 1158 ts_tablet_manager.cc:1397] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20250109 20:52:53.137565 1158 raft_consensus.cc:357] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.137753 1158 raft_consensus.cc:383] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.137838 1158 raft_consensus.cc:738] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.140056 1157 ts_tablet_manager.cc:1905] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.138075 1158 consensus_queue.cc:260] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.143344 1158 ts_tablet_manager.cc:1428] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:52:53.143616 1158 tablet_bootstrap.cc:492] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.145929 1158 tablet_bootstrap.cc:654] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.147456 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c368e39a840a4eceaf17180255fda3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.150722 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03310d8c1ff4b3a82dd0ce35992a319. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.151235 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d3f97679c8408b9248b126a2218313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.151669 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9c79ecf9db554b50b0a61325275b9bee (DEFAULT_TABLE table=default.Table_88ff1094b93e4aedb7f622f1d97becaf [id=32c9cb3f999b4a45973ba2b5e8789b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.152047 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c79ecf9db554b50b0a61325275b9bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.153613 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4a99e7e38d8a40268faa7241d72850d6 (DEFAULT_TABLE table=default.Table_6c4a9feb3c4b4f86a11bb8ff982d4304 [id=475a6b5c9eca4539b499ceb3ac280b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.153956 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a99e7e38d8a40268faa7241d72850d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.159911 896 tablet_service.cc:1467] Processing CreateTablet for tablet 9c79ecf9db554b50b0a61325275b9bee (DEFAULT_TABLE table=default.Table_88ff1094b93e4aedb7f622f1d97becaf [id=32c9cb3f999b4a45973ba2b5e8789b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.160257 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c79ecf9db554b50b0a61325275b9bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.166904 880 tablet_service.cc:1467] Processing CreateTablet for tablet 4a99e7e38d8a40268faa7241d72850d6 (DEFAULT_TABLE table=default.Table_6c4a9feb3c4b4f86a11bb8ff982d4304 [id=475a6b5c9eca4539b499ceb3ac280b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.167407 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a99e7e38d8a40268faa7241d72850d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.168038 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6630c808e354ecdb168332d5c3ea8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.174266 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 9c79ecf9db554b50b0a61325275b9bee (DEFAULT_TABLE table=default.Table_88ff1094b93e4aedb7f622f1d97becaf [id=32c9cb3f999b4a45973ba2b5e8789b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.174757 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c79ecf9db554b50b0a61325275b9bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.182334 758 tablet_service.cc:1467] Processing CreateTablet for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 (DEFAULT_TABLE table=default.Table_6223d4bbeb7a4deebddb2936e92224c0 [id=f88d49eba7374387915b697342d8ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.182746 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f77f35085ff43a5b7fb6cce7e7a3409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.186659 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 2be32f590fd844b2bbff91488a5390fc (DEFAULT_TABLE table=default.Table_e4be594b7e46489d83d7c7f8bbbed681 [id=4b8a5751a09e43fb8bde0d5b25886f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.186892 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 575f3cb9f61143ab949fa85f47edcafb (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.187053 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2be32f590fd844b2bbff91488a5390fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.186659 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 (DEFAULT_TABLE table=default.Table_6223d4bbeb7a4deebddb2936e92224c0 [id=f88d49eba7374387915b697342d8ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.187403 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f77f35085ff43a5b7fb6cce7e7a3409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.188848 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575f3cb9f61143ab949fa85f47edcafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.188997 1028 tablet_service.cc:1467] Processing CreateTablet for tablet bcd3219734314ae1ad03968fdf9c9cd0 (DEFAULT_TABLE table=default.Table_4cf7db6076db4bc4a5b7527f7b0a41cc [id=ca4a28a7f83642d6b9c4206605c3c835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.189285 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd3219734314ae1ad03968fdf9c9cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.195446 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 4870a667cd3e410590923c9e2497af54 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:53.197046 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 (DEFAULT_TABLE table=default.Table_169be780c136447db26907a86d29867a [id=b772ba6a77ce40df89090e51d031534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.197372 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.200630 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bf29c4a64f7b4d49af314ad6211982b1" } modify_external_catalogs: true
I20250109 20:52:53.200868 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bf29c4a64f7b4d49af314ad6211982b1" } modify_external_catalogs: true
I20250109 20:52:53.201539 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet e64207928ef141118e7fc8fe7bb2379b (table default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]) successfully deleted
I20250109 20:52:53.202188 1166 tablet_replica.cc:331] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:53.202374 1166 raft_consensus.cc:2238] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:53.202385 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2ca83a49523b45be9ac7556b40746a83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ca83a49523b45be9ac7556b40746a83 already in progress: creating tablet
I20250109 20:52:53.202543 1166 raft_consensus.cc:2267] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:53.202593 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 687db3e734af47339c2546970a71a539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 687db3e734af47339c2546970a71a539 already in progress: creating tablet
W20250109 20:52:53.202802 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 189cd8079a9d4ca6b1703f4361d5ec5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 189cd8079a9d4ca6b1703f4361d5ec5d already in progress: creating tablet
W20250109 20:52:53.203022 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ed00f7954143c9ba9bf65d8a8a68d2 already in progress: creating tablet
W20250109 20:52:53.203241 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72cc0bbbb5584f14ba4d6c62211a12c5 already in progress: creating tablet
I20250109 20:52:53.203578 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_19363d9282fe42228ab784489a2b9a17" } new_table_name: "default.Table_3254b432932b4d9bb83cb8a2f26e5514" modify_external_catalogs: true
I20250109 20:52:53.205173 1176 tablet_bootstrap.cc:492] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.205096 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_546faac34e4d4250b0845e4886d01377" } modify_external_catalogs: true
I20250109 20:52:53.205278 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_546faac34e4d4250b0845e4886d01377" } modify_external_catalogs: true
I20250109 20:52:53.205323 1176 ts_tablet_manager.cc:1397] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.124s user 0.002s sys 0.003s
I20250109 20:52:53.206032 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3bc3ade0c5964f94802bd73bd9e69f21" } new_table_name: "default.Table_362d78bbb5d74965bc37f992157c663b" modify_external_catalogs: true
I20250109 20:52:53.206079 1176 raft_consensus.cc:357] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.206271 1176 raft_consensus.cc:383] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.206346 1176 raft_consensus.cc:738] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.206570 1176 consensus_queue.cc:260] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.206920 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_88ff1094b93e4aedb7f622f1d97becaf" } new_table_name: "default.Table_d0b623301b834635896597ef9d332c21" modify_external_catalogs: true
I20250109 20:52:53.207039 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ad0a7364cd0d48caa4d583519597f8fe" } modify_external_catalogs: true
I20250109 20:52:53.207145 1176 ts_tablet_manager.cc:1428] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:52:53.207180 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ad0a7364cd0d48caa4d583519597f8fe" } modify_external_catalogs: true
I20250109 20:52:53.208068 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e6adec6051bc4812bb9fbc119614d604" } modify_external_catalogs: true
I20250109 20:52:53.208235 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e6adec6051bc4812bb9fbc119614d604" } modify_external_catalogs: true
I20250109 20:52:53.213397 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5377c27f242f4aaf8ef9d0281640a856" } modify_external_catalogs: true
I20250109 20:52:53.213582 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5377c27f242f4aaf8ef9d0281640a856" } modify_external_catalogs: true
I20250109 20:52:53.215400 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 46df992009994d9e879109ba14bb06c0 (DEFAULT_TABLE table=default.Table_5fef44f0700046e0ba377025bb4d4c78 [id=96d115ee7e824e2eb0d56ff8787ea3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.215543 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_bc43af02a045473cac4e0b21c2dfc318 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 1 got 0
I20250109 20:52:53.217070 1160 ts_tablet_manager.cc:1918] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.218848 888 tablet_service.cc:1467] Processing CreateTablet for tablet bcd3219734314ae1ad03968fdf9c9cd0 (DEFAULT_TABLE table=default.Table_4cf7db6076db4bc4a5b7527f7b0a41cc [id=ca4a28a7f83642d6b9c4206605c3c835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.219039 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4a99e7e38d8a40268faa7241d72850d6 (DEFAULT_TABLE table=default.Table_6c4a9feb3c4b4f86a11bb8ff982d4304 [id=475a6b5c9eca4539b499ceb3ac280b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.219273 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 with delete_type TABLET_DATA_DELETED (replaced by tablet 6619b3dce3ed4325a4871c3d40b42c9d) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:53.219905 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd3219734314ae1ad03968fdf9c9cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.220137 1166 ts_tablet_manager.cc:1905] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.222473 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f32d9ff86dad43228e89600903a000f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.223055 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f32d9ff86dad43228e89600903a000f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.223125 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46df992009994d9e879109ba14bb06c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.224321 895 tablet_service.cc:1467] Processing CreateTablet for tablet 2be32f590fd844b2bbff91488a5390fc (DEFAULT_TABLE table=default.Table_e4be594b7e46489d83d7c7f8bbbed681 [id=4b8a5751a09e43fb8bde0d5b25886f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.224684 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2be32f590fd844b2bbff91488a5390fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.224926 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a99e7e38d8a40268faa7241d72850d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.229568 898 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 (DEFAULT_TABLE table=default.Table_169be780c136447db26907a86d29867a [id=b772ba6a77ce40df89090e51d031534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.230227 1176 tablet_bootstrap.cc:492] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.231001 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcadec0c3184feaad4a883e5588f310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:53.231755 1176 tablet_bootstrap.cc:654] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.235020 757 tablet_service.cc:1467] Processing CreateTablet for tablet 46df992009994d9e879109ba14bb06c0 (DEFAULT_TABLE table=default.Table_5fef44f0700046e0ba377025bb4d4c78 [id=96d115ee7e824e2eb0d56ff8787ea3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.235479 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.235440 751 tablet_service.cc:1467] Processing CreateTablet for tablet 575f3cb9f61143ab949fa85f47edcafb (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.235693 749 tablet_service.cc:1467] Processing CreateTablet for tablet bcd3219734314ae1ad03968fdf9c9cd0 (DEFAULT_TABLE table=default.Table_4cf7db6076db4bc4a5b7527f7b0a41cc [id=ca4a28a7f83642d6b9c4206605c3c835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.236011 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_532e9f42a7084d3c9037828c563a3f7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.236552 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_532e9f42a7084d3c9037828c563a3f7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.237059 1160 log.cc:1198] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/49c6f1833ff1453f90a6ec6435248a02
I20250109 20:52:53.237767 1166 ts_tablet_manager.cc:1918] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.237877 1166 log.cc:1198] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/46d4cad7594346dfafdee8e4c49d74e6
I20250109 20:52:53.237846 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 (DEFAULT_TABLE table=default.Table_6223d4bbeb7a4deebddb2936e92224c0 [id=f88d49eba7374387915b697342d8ddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.238204 1166 ts_tablet_manager.cc:1939] T 46d4cad7594346dfafdee8e4c49d74e6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:53.238379 885 tablet_service.cc:1467] Processing CreateTablet for tablet 575f3cb9f61143ab949fa85f47edcafb (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.239398 887 tablet_service.cc:1467] Processing CreateTablet for tablet 46df992009994d9e879109ba14bb06c0 (DEFAULT_TABLE table=default.Table_5fef44f0700046e0ba377025bb4d4c78 [id=96d115ee7e824e2eb0d56ff8787ea3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.240234 883 tablet_service.cc:1467] Processing CreateTablet for tablet 09e412e3a03f42c083032fdf3854ef39 (DEFAULT_TABLE table=default.Table_c4ae127254154c808ee79dd8e20f184a [id=c7ee2e1c75834a08a3685a53593fe8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.242439 890 tablet_service.cc:1467] Processing CreateTablet for tablet 0f68bc0a564c4210bb29172a8759ebf3 (DEFAULT_TABLE table=default.Table_3b054fb299c84cf49c242e3ccb44daf8 [id=a8d9617e1d1d4488abe06c7065bfee48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.242733 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2797e1df3d12476f89fb5155523c3935
I20250109 20:52:53.243116 756 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 (DEFAULT_TABLE table=default.Table_169be780c136447db26907a86d29867a [id=b772ba6a77ce40df89090e51d031534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.243388 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46df992009994d9e879109ba14bb06c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.243364 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6e345f747be74c0c9cfb28cf6ff90e93" } new_table_name: "default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0" modify_external_catalogs: true
I20250109 20:52:53.243594 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575f3cb9f61143ab949fa85f47edcafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.243678 742 tablet_service.cc:1467] Processing CreateTablet for tablet 2be32f590fd844b2bbff91488a5390fc (DEFAULT_TABLE table=default.Table_e4be594b7e46489d83d7c7f8bbbed681 [id=4b8a5751a09e43fb8bde0d5b25886f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.244414 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2be32f590fd844b2bbff91488a5390fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.244972 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd3219734314ae1ad03968fdf9c9cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.245132 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet bfa2ae1488744585805590987eee7b59
W20250109 20:52:53.245329 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9629fe30752f49478afad9b93dc235eb consensus state has no leader
I20250109 20:52:53.245520 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4a5f5d985f414ece9c3a1e4962abe0b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:53.245797 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_69bb01d4e8aa4563a727e44e5f6b3c20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.246044 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a5f5d985f414ece9c3a1e4962abe0b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:53.246210 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69bb01d4e8aa4563a727e44e5f6b3c20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.247123 1176 tablet_bootstrap.cc:492] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.247251 1176 ts_tablet_manager.cc:1397] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
W20250109 20:52:53.247318 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c79ecf9db554b50b0a61325275b9bee consensus state has no leader
I20250109 20:52:53.247524 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 540531463c184b9abf226d7487939e60 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:53.247918 1176 raft_consensus.cc:357] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.248000 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797e1df3d12476f89fb5155523c3935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:53.248085 1176 raft_consensus.cc:383] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.248148 1176 raft_consensus.cc:738] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.248143 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 678ab724db1a42e2b23bbe6cd7652db7
I20250109 20:52:53.248327 899 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac6f8baab5040abbe36e185a8052c6e (DEFAULT_TABLE table=default.Table_b43e8c35fcdf4973ada77b9c4ad44ba0 [id=414c15fe1d1845858a0839cb4fa668a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.248765 889 tablet_service.cc:1467] Processing CreateTablet for tablet a279331730c7482580650c24365c4cba (DEFAULT_TABLE table=default.Table_7d9492ed2e5b44ada92f6fa5faa03ab9 [id=4bc4c77edc4a468f9c76d7a65b6badec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.248919 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e0f67c4294f8499496db4e82db8c3bf0
I20250109 20:52:53.248844 1176 consensus_queue.cc:260] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.249465 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 418e1bd414914b21a233c3099e2057da
W20250109 20:52:53.250119 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f953f43877c4a738ffd8255afbf8275 consensus state has no leader
I20250109 20:52:53.250602 1160 ts_tablet_manager.cc:1939] T 49c6f1833ff1453f90a6ec6435248a02 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:53.251116 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 678ab724db1a42e2b23bbe6cd7652db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:53.251574 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.251598 894 tablet_service.cc:1467] Processing CreateTablet for tablet 3464ea8be70f494183c1395c9b2b0784 (DEFAULT_TABLE table=default.Table_6e96bb9287564f70abcd0b9f9865e77f [id=69a3c4ad65704ac5a50690d19330171a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.251832 882 tablet_service.cc:1467] Processing CreateTablet for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 (DEFAULT_TABLE table=default.Table_3beb193926bd4164a21137f32bf717ac [id=c26de09aefb246f3a7a8b34c1e38f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.252141 1176 ts_tablet_manager.cc:1428] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:53.252203 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 418e1bd414914b21a233c3099e2057da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:53.252403 1176 tablet_bootstrap.cc:492] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.252743 886 tablet_service.cc:1467] Processing CreateTablet for tablet d4b939cdea1c4719b9b3918aabbdfc0e (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.252934 881 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:53.252341 884 tablet_service.cc:1467] Processing CreateTablet for tablet 42d3318891db42e68836b9fd3b507de4 (DEFAULT_TABLE table=default.Table_d664d9d1e75b4d728aadaa5ef1103542 [id=eafc9479d4524867bf1199e70fc23bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.253984 1176 tablet_bootstrap.cc:654] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.256486 881 tablet_service.cc:1467] Processing CreateTablet for tablet 5c918f5432574dc38d28609c918131fb (DEFAULT_TABLE table=default.Table_6bcb56e269714b399e7d1fbc2e2c290e [id=bafe557245e043e583947c62bf99b56c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.256552 1027 tablet_service.cc:1467] Processing CreateTablet for tablet f3c2769694f14ddd8d2e351974949b77 (DEFAULT_TABLE table=default.Table_c271524e66094f2e9f698e6e174de073 [id=60061719478c4a6eb828cda73185dc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.256929 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c2769694f14ddd8d2e351974949b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.257404 1016 tablet_service.cc:1467] Processing CreateTablet for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 (DEFAULT_TABLE table=default.Table_3beb193926bd4164a21137f32bf717ac [id=c26de09aefb246f3a7a8b34c1e38f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.257752 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab64a4db7fa241d7a0a4fb73622e6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.258013 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_bc43af02a045473cac4e0b21c2dfc318 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 1 got 0
I20250109 20:52:53.259790 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 46d4cad7594346dfafdee8e4c49d74e6 (table default.Table_49b7725a1f7647cd9b136c55bd8ed301 [id=ccd17bbcb44f49a3b2eae73c6ace6c4c]) successfully deleted
I20250109 20:52:53.261063 1033 tablet_service.cc:1467] Processing CreateTablet for tablet d4b939cdea1c4719b9b3918aabbdfc0e (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.261351 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b939cdea1c4719b9b3918aabbdfc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.261300 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 09e412e3a03f42c083032fdf3854ef39 (DEFAULT_TABLE table=default.Table_c4ae127254154c808ee79dd8e20f184a [id=c7ee2e1c75834a08a3685a53593fe8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.261556 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e412e3a03f42c083032fdf3854ef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.261631 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797e1df3d12476f89fb5155523c3935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:53.266758 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_acafc59f57e24e4e98ae6b7b3a50fccb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.267237 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acafc59f57e24e4e98ae6b7b3a50fccb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.268417 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_503a302a104946dba7674a2aba9dba11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.268873 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_503a302a104946dba7674a2aba9dba11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.269141 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac6f8baab5040abbe36e185a8052c6e (DEFAULT_TABLE table=default.Table_b43e8c35fcdf4973ada77b9c4ad44ba0 [id=414c15fe1d1845858a0839cb4fa668a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.269454 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac6f8baab5040abbe36e185a8052c6e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:53.269613 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f407b76780974cf7b3635fa1bf1585ec (table default.Table_5e53bf0c9fce4d04a077919ffb080698 [id=32d713e3cdee403cb9193769815a07fa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f407b76780974cf7b3635fa1bf1585ec consensus state has no leader
I20250109 20:52:53.269997 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4ddee41d2e28400c93caa1462c016e5b" } new_table_name: "default.Table_f7662dfa61fe4100a214839f938d4a58" modify_external_catalogs: true
I20250109 20:52:53.271167 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_43fc99f8a3c8473fa4c52881041e22a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.271652 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43fc99f8a3c8473fa4c52881041e22a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.272359 891 tablet_service.cc:1467] Processing CreateTablet for tablet be05de223c3946c6b8a4899cf696fe26 (DEFAULT_TABLE table=default.Table_cede232c02904c21a0d46c338205fa14 [id=3bdc7c0be0d14815a92667bde1011e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.273288 892 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 (DEFAULT_TABLE table=default.Table_9939ee8672474bc0bcd631684079d0c3 [id=33883599d93e4b198768a844ea4b05f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.273696 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 418e1bd414914b21a233c3099e2057da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:53.274338 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 678ab724db1a42e2b23bbe6cd7652db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:53.274974 1027 tablet_service.cc:1467] Processing CreateTablet for tablet a279331730c7482580650c24365c4cba (DEFAULT_TABLE table=default.Table_7d9492ed2e5b44ada92f6fa5faa03ab9 [id=4bc4c77edc4a468f9c76d7a65b6badec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.275911 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a279331730c7482580650c24365c4cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.276346 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 49c6f1833ff1453f90a6ec6435248a02 (table default.Table_d59d31f7bc514ff38a99d27cbe8170f9 [id=5514c15f7b8c406288bec6394be1d99b]) successfully deleted
I20250109 20:52:53.277348 744 tablet_service.cc:1467] Processing CreateTablet for tablet f3c2769694f14ddd8d2e351974949b77 (DEFAULT_TABLE table=default.Table_c271524e66094f2e9f698e6e174de073 [id=60061719478c4a6eb828cda73185dc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.277379 754 tablet_service.cc:1467] Processing CreateTablet for tablet 0f68bc0a564c4210bb29172a8759ebf3 (DEFAULT_TABLE table=default.Table_3b054fb299c84cf49c242e3ccb44daf8 [id=a8d9617e1d1d4488abe06c7065bfee48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.277667 748 tablet_service.cc:1467] Processing CreateTablet for tablet 09e412e3a03f42c083032fdf3854ef39 (DEFAULT_TABLE table=default.Table_c4ae127254154c808ee79dd8e20f184a [id=c7ee2e1c75834a08a3685a53593fe8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.277760 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3464ea8be70f494183c1395c9b2b0784 (DEFAULT_TABLE table=default.Table_6e96bb9287564f70abcd0b9f9865e77f [id=69a3c4ad65704ac5a50690d19330171a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.278034 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3464ea8be70f494183c1395c9b2b0784. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.278082 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c2769694f14ddd8d2e351974949b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.278277 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e412e3a03f42c083032fdf3854ef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.278497 752 tablet_service.cc:1467] Processing CreateTablet for tablet 42d3318891db42e68836b9fd3b507de4 (DEFAULT_TABLE table=default.Table_d664d9d1e75b4d728aadaa5ef1103542 [id=eafc9479d4524867bf1199e70fc23bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.278604 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f68bc0a564c4210bb29172a8759ebf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.279098 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d3318891db42e68836b9fd3b507de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.279939 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797e1df3d12476f89fb5155523c3935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.280151 753 tablet_service.cc:1467] Processing CreateTablet for tablet d4b939cdea1c4719b9b3918aabbdfc0e (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.280457 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b939cdea1c4719b9b3918aabbdfc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.284449 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 0f68bc0a564c4210bb29172a8759ebf3 (DEFAULT_TABLE table=default.Table_3b054fb299c84cf49c242e3ccb44daf8 [id=a8d9617e1d1d4488abe06c7065bfee48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.284449 746 tablet_service.cc:1467] Processing CreateTablet for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 (DEFAULT_TABLE table=default.Table_3beb193926bd4164a21137f32bf717ac [id=c26de09aefb246f3a7a8b34c1e38f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.284996 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab64a4db7fa241d7a0a4fb73622e6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.285375 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f68bc0a564c4210bb29172a8759ebf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.285820 755 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac6f8baab5040abbe36e185a8052c6e (DEFAULT_TABLE table=default.Table_b43e8c35fcdf4973ada77b9c4ad44ba0 [id=414c15fe1d1845858a0839cb4fa668a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.286131 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac6f8baab5040abbe36e185a8052c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.288021 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:53.288195 745 tablet_service.cc:1467] Processing CreateTablet for tablet a279331730c7482580650c24365c4cba (DEFAULT_TABLE table=default.Table_7d9492ed2e5b44ada92f6fa5faa03ab9 [id=4bc4c77edc4a468f9c76d7a65b6badec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.288520 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a279331730c7482580650c24365c4cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.288024 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 5c918f5432574dc38d28609c918131fb (DEFAULT_TABLE table=default.Table_6bcb56e269714b399e7d1fbc2e2c290e [id=bafe557245e043e583947c62bf99b56c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.287981 1032 tablet_service.cc:1467] Processing CreateTablet for tablet be05de223c3946c6b8a4899cf696fe26 (DEFAULT_TABLE table=default.Table_cede232c02904c21a0d46c338205fa14 [id=3bdc7c0be0d14815a92667bde1011e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.289126 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 678ab724db1a42e2b23bbe6cd7652db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.289408 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c918f5432574dc38d28609c918131fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.290138 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 42d3318891db42e68836b9fd3b507de4 (DEFAULT_TABLE table=default.Table_d664d9d1e75b4d728aadaa5ef1103542 [id=eafc9479d4524867bf1199e70fc23bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.290369 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be05de223c3946c6b8a4899cf696fe26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.293608 1176 tablet_bootstrap.cc:492] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.293658 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 (DEFAULT_TABLE table=default.Table_9939ee8672474bc0bcd631684079d0c3 [id=33883599d93e4b198768a844ea4b05f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.294147 1176 ts_tablet_manager.cc:1397] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250109 20:52:53.294245 741 tablet_service.cc:1467] Processing CreateTablet for tablet 3464ea8be70f494183c1395c9b2b0784 (DEFAULT_TABLE table=default.Table_6e96bb9287564f70abcd0b9f9865e77f [id=69a3c4ad65704ac5a50690d19330171a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.294337 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 418e1bd414914b21a233c3099e2057da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.294720 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d3318891db42e68836b9fd3b507de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.299832 1160 tablet_replica.cc:331] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:53.299854 1176 raft_consensus.cc:357] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.300130 1160 raft_consensus.cc:2238] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.300515 1160 raft_consensus.cc:2267] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.301499 1160 ts_tablet_manager.cc:1905] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.301661 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f77f35085ff43a5b7fb6cce7e7a3409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.301864 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d3318891db42e68836b9fd3b507de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.306393 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c918f5432574dc38d28609c918131fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.307981 1176 raft_consensus.cc:383] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.308087 1176 raft_consensus.cc:738] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.308703 1176 consensus_queue.cc:260] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.309329 1176 ts_tablet_manager.cc:1428] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250109 20:52:53.310135 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.313550 743 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.294262 750 tablet_service.cc:1467] Processing CreateTablet for tablet 5c918f5432574dc38d28609c918131fb (DEFAULT_TABLE table=default.Table_6bcb56e269714b399e7d1fbc2e2c290e [id=bafe557245e043e583947c62bf99b56c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.315714 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3464ea8be70f494183c1395c9b2b0784. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.316977 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:53.320554 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 80da78917f0b4931ab1705371c1fe5dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80da78917f0b4931ab1705371c1fe5dd already in progress: creating tablet
I20250109 20:52:53.320987 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a279331730c7482580650c24365c4cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.323261 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e412e3a03f42c083032fdf3854ef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.327343 1176 tablet_bootstrap.cc:492] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.328899 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ac6f8baab5040abbe36e185a8052c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.329402 1176 tablet_bootstrap.cc:654] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.332834 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3464ea8be70f494183c1395c9b2b0784. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.338163 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
W20250109 20:52:53.342367 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 126311us
W20250109 20:52:53.350572 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cda300a7e58a44339c3bdb2e62a9f384 consensus state has no leader
W20250109 20:52:53.354363 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c5e9b31b27949c2948cb64b6124ee72 consensus state has no leader
I20250109 20:52:53.357498 1176 tablet_bootstrap.cc:492] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.357628 1176 ts_tablet_manager.cc:1397] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250109 20:52:53.358024 1166 tablet_replica.cc:331] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:53.358268 1166 raft_consensus.cc:2238] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.358341 1176 raft_consensus.cc:357] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.358469 1166 raft_consensus.cc:2267] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.358505 1176 raft_consensus.cc:383] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.358835 1176 raft_consensus.cc:738] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.359090 1176 consensus_queue.cc:260] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.360790 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:53.361011 1176 ts_tablet_manager.cc:1428] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:53.361254 1176 tablet_bootstrap.cc:492] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.363296 1176 tablet_bootstrap.cc:654] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.364531 1166 ts_tablet_manager.cc:1905] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.369665 1678 raft_consensus.cc:491] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.369813 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 540531463c184b9abf226d7487939e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.370978 1176 tablet_bootstrap.cc:492] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.371102 1176 ts_tablet_manager.cc:1397] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250109 20:52:53.371685 1176 raft_consensus.cc:357] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.371851 1176 raft_consensus.cc:383] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.371908 1176 raft_consensus.cc:738] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.372112 1176 consensus_queue.cc:260] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.373050 1176 ts_tablet_manager.cc:1428] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:52:53.369863 1678 raft_consensus.cc:513] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.375845 1176 tablet_bootstrap.cc:492] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.376320 1678 leader_election.cc:290] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.376477 1678 consensus_queue.cc:1035] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:53.377241 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc2c7b12f945ea8307a187e1553226" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.377398 1176 tablet_bootstrap.cc:654] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:53.377933 712 leader_election.cc:343] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.378057 1166 ts_tablet_manager.cc:1918] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.383047 1166 log.cc:1198] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f5a5815d9ef54e3e959f151ed08ae812
I20250109 20:52:53.383519 1166 ts_tablet_manager.cc:1939] T f5a5815d9ef54e3e959f151ed08ae812 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:53.384307 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c918f5432574dc38d28609c918131fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.385540 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f5a5815d9ef54e3e959f151ed08ae812 (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:52:53.385990 1166 tablet_replica.cc:331] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:53.386261 1166 raft_consensus.cc:2238] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.386524 1166 raft_consensus.cc:2267] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.393502 1166 ts_tablet_manager.cc:1905] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.393955 739 tablet_service.cc:1467] Processing CreateTablet for tablet be05de223c3946c6b8a4899cf696fe26 (DEFAULT_TABLE table=default.Table_cede232c02904c21a0d46c338205fa14 [id=3bdc7c0be0d14815a92667bde1011e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.395179 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b77c8d6adbf44375aff2782a83643255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.398036 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_00dd367ba48b4c8c93d19c4ec1d7017c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.398564 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00dd367ba48b4c8c93d19c4ec1d7017c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.399186 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f68bc0a564c4210bb29172a8759ebf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.400095 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b939cdea1c4719b9b3918aabbdfc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.400249 757 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 (DEFAULT_TABLE table=default.Table_9939ee8672474bc0bcd631684079d0c3 [id=33883599d93e4b198768a844ea4b05f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.401978 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d8074d797bd94aaea619f6772e0c644c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.402410 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8074d797bd94aaea619f6772e0c644c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.403025 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be05de223c3946c6b8a4899cf696fe26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.405622 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46df992009994d9e879109ba14bb06c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.406275 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab64a4db7fa241d7a0a4fb73622e6eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.407627 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575f3cb9f61143ab949fa85f47edcafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.411412 1158 tablet_bootstrap.cc:492] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.411967 896 tablet_service.cc:1467] Processing CreateTablet for tablet f3c2769694f14ddd8d2e351974949b77 (DEFAULT_TABLE table=default.Table_c271524e66094f2e9f698e6e174de073 [id=60061719478c4a6eb828cda73185dc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:53.412366 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c2769694f14ddd8d2e351974949b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.414183 454 master_service.cc:946] ReplaceTablet: received request to replace tablet b68d23fb2df743f4afa2682f6d2e3efa from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:53.414563 1158 ts_tablet_manager.cc:1397] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.268s user 0.018s sys 0.047s
I20250109 20:52:53.416713 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be05de223c3946c6b8a4899cf696fe26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.416783 1166 ts_tablet_manager.cc:1918] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.417631 1158 raft_consensus.cc:357] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.418040 1166 log.cc:1198] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/e4c367a8ca184dfa9e4b11094ba76140
I20250109 20:52:53.418823 1166 ts_tablet_manager.cc:1939] T e4c367a8ca184dfa9e4b11094ba76140 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:53.422277 1166 tablet_replica.cc:331] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:53.422592 1166 raft_consensus.cc:2238] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.422691 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet e4c367a8ca184dfa9e4b11094ba76140 (table default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]) successfully deleted
I20250109 20:52:53.422847 1166 raft_consensus.cc:2267] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.423195 1158 raft_consensus.cc:383] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.423804 1176 tablet_bootstrap.cc:492] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.423962 1176 ts_tablet_manager.cc:1397] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.010s
I20250109 20:52:53.424340 1166 ts_tablet_manager.cc:1905] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.424988 1176 raft_consensus.cc:357] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.425271 1176 raft_consensus.cc:383] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.425383 1176 raft_consensus.cc:738] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.425598 1176 consensus_queue.cc:260] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.426107 1176 ts_tablet_manager.cc:1428] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:53.426386 1176 tablet_bootstrap.cc:492] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.427923 1176 tablet_bootstrap.cc:654] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.435870 1158 raft_consensus.cc:738] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.440846 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:53.441627 1158 consensus_queue.cc:260] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.442847 1165 tablet_bootstrap.cc:492] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:53.443028 1165 ts_tablet_manager.cc:1397] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.373s user 0.012s sys 0.009s
I20250109 20:52:53.453240 1166 ts_tablet_manager.cc:1918] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.454483 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bbdaad91fc3541f79444f6adea0e9e4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:53.455005 1166 log.cc:1198] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/fbcadec0c3184feaad4a883e5588f310
W20250109 20:52:53.455147 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbdaad91fc3541f79444f6adea0e9e4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.455412 1166 ts_tablet_manager.cc:1939] T fbcadec0c3184feaad4a883e5588f310 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:53.455868 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe" } modify_external_catalogs: true
I20250109 20:52:53.456028 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe" } modify_external_catalogs: true
I20250109 20:52:53.456496 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet fbcadec0c3184feaad4a883e5588f310 (table default.Table_e951676901de4adab1fb93a2d265da4f [id=97659ab8e1774179b7854d0985954b0b]) successfully deleted
I20250109 20:52:53.456517 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_036a876d87ae4027b9002e151ed56074" } modify_external_catalogs: true
I20250109 20:52:53.456647 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_036a876d87ae4027b9002e151ed56074" } modify_external_catalogs: true
I20250109 20:52:53.456946 1771 raft_consensus.cc:491] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:52:53.457044 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2797e1df3d12476f89fb5155523c3935 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2797e1df3d12476f89fb5155523c3935 already in progress: creating tablet
I20250109 20:52:53.457093 1771 raft_consensus.cc:513] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.457218 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8ed167dde7b94790b7731c10aa30628c" } modify_external_catalogs: true
I20250109 20:52:53.457345 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8ed167dde7b94790b7731c10aa30628c" } modify_external_catalogs: true
W20250109 20:52:53.457243 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 418e1bd414914b21a233c3099e2057da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 418e1bd414914b21a233c3099e2057da already in progress: creating tablet
I20250109 20:52:53.457693 1771 leader_election.cc:290] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.458060 1158 ts_tablet_manager.cc:1428] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20250109 20:52:53.458235 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b06e8616df461d89c58d414fc0cfbd" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:53.458518 1053 raft_consensus.cc:2388] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d473955919435cb679edd5f8afe2c2 in current term 1: Already voted for candidate 09833ea46b7141d58c3899dace69e49f in this term.
I20250109 20:52:53.459625 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_362d78bbb5d74965bc37f992157c663b" } new_table_name: "default.Table_40c6cee0e5254b01b5329858620643fb" modify_external_catalogs: true
I20250109 20:52:53.460615 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6dfedfbc9bba4dc8a4fb00f68b3ba689" } new_table_name: "default.Table_013e0df3d370438496fdffef9ce40df2" modify_external_catalogs: true
W20250109 20:52:53.461541 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 678ab724db1a42e2b23bbe6cd7652db7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 678ab724db1a42e2b23bbe6cd7652db7 already in progress: creating tablet
W20250109 20:52:53.461755 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e0f67c4294f8499496db4e82db8c3bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f67c4294f8499496db4e82db8c3bf0 already in progress: creating tablet
I20250109 20:52:53.461906 1157 ts_tablet_manager.cc:1918] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.462018 1157 log.cc:1198] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/c03524194a7b4e34a20424091e308ac4
I20250109 20:52:53.467201 1158 tablet_bootstrap.cc:492] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.467816 1157 ts_tablet_manager.cc:1939] T c03524194a7b4e34a20424091e308ac4 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:53.471328 1160 ts_tablet_manager.cc:1918] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250109 20:52:53.471432 1160 log.cc:1198] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/9c7ed9e8c921491c9d911d662f500451
I20250109 20:52:53.471882 1160 ts_tablet_manager.cc:1939] T 9c7ed9e8c921491c9d911d662f500451 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:53.472313 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 9c7ed9e8c921491c9d911d662f500451 (table default.Table_76d70836419947c78b06922a4fe21a2c [id=53718e55e4344456851f408909cc8170]) successfully deleted
I20250109 20:52:53.472630 1165 raft_consensus.cc:357] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.472915 1165 raft_consensus.cc:383] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.472924 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b06e8616df461d89c58d414fc0cfbd" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:53.473677 1165 raft_consensus.cc:738] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.473754 854 leader_election.cc:304] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:53.473990 1165 consensus_queue.cc:260] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.474188 1771 raft_consensus.cc:3054] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:53.476596 1157 tablet_replica.cc:331] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:53.476830 1157 raft_consensus.cc:2238] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.477000 1771 raft_consensus.cc:2743] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:53.477027 1157 raft_consensus.cc:2267] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.477653 1157 ts_tablet_manager.cc:1905] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.478953 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet c03524194a7b4e34a20424091e308ac4 (table default.Table_df185d0c630644438a9e8cc5fe14446b [id=d89b416c6f8d4417a6ccb26b443b2ef4]) successfully deleted
I20250109 20:52:53.482059 1176 tablet_bootstrap.cc:492] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.482257 1176 ts_tablet_manager.cc:1397] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.010s
I20250109 20:52:53.483125 1176 raft_consensus.cc:357] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.483462 1176 raft_consensus.cc:383] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.483584 1176 raft_consensus.cc:738] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.483934 1176 consensus_queue.cc:260] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.484534 1176 ts_tablet_manager.cc:1428] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:52:53.484997 1176 tablet_bootstrap.cc:492] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.486563 1176 tablet_bootstrap.cc:654] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.495362 1158 tablet_bootstrap.cc:654] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:53.496270 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 122518us
I20250109 20:52:53.497119 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc2c7b12f945ea8307a187e1553226" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:53.497409 1053 raft_consensus.cc:2463] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:53.505721 1803 tablet.cc:1721] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:53.515789 1157 ts_tablet_manager.cc:1918] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.516099 1157 log.cc:1198] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6fb563bd6802473fafedd08bb10d15c1
I20250109 20:52:53.516469 1157 ts_tablet_manager.cc:1939] T 6fb563bd6802473fafedd08bb10d15c1 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:53.516726 1158 tablet_bootstrap.cc:492] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.516848 1158 ts_tablet_manager.cc:1397] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.003s
I20250109 20:52:53.517446 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6fb563bd6802473fafedd08bb10d15c1 (table default.Table_d2ee28b720884b02a77177a6c829d85a [id=dccf417b39fc4d2d87812e340ecd842e]) successfully deleted
I20250109 20:52:53.517520 1158 raft_consensus.cc:357] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.517683 1158 raft_consensus.cc:383] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.517760 1158 raft_consensus.cc:738] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:53.518082 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2797e1df3d12476f89fb5155523c3935 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2797e1df3d12476f89fb5155523c3935 already in progress: creating tablet
I20250109 20:52:53.517999 1158 consensus_queue.cc:260] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.518553 1158 ts_tablet_manager.cc:1428] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250109 20:52:53.518627 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 678ab724db1a42e2b23bbe6cd7652db7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 678ab724db1a42e2b23bbe6cd7652db7 already in progress: creating tablet
I20250109 20:52:53.519069 1158 tablet_bootstrap.cc:492] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.520699 1158 tablet_bootstrap.cc:654] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.523064 713 leader_election.cc:304] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.524917 1678 raft_consensus.cc:2798] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:53.525034 1678 raft_consensus.cc:491] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:53.525162 1678 raft_consensus.cc:3054] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:53.527649 1158 tablet_bootstrap.cc:492] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.527783 1158 ts_tablet_manager.cc:1397] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20250109 20:52:53.528474 1158 raft_consensus.cc:357] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.528412 1678 raft_consensus.cc:513] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:53.529369 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 418e1bd414914b21a233c3099e2057da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 418e1bd414914b21a233c3099e2057da already in progress: creating tablet
W20250109 20:52:53.529981 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e0f67c4294f8499496db4e82db8c3bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f67c4294f8499496db4e82db8c3bf0 already in progress: creating tablet
I20250109 20:52:53.530571 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc2c7b12f945ea8307a187e1553226" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:53.531271 712 leader_election.cc:343] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.531903 1157 tablet_replica.cc:331] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:53.532140 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc2c7b12f945ea8307a187e1553226" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:53.532364 1055 raft_consensus.cc:3054] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:53.529249 1158 raft_consensus.cc:383] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.532771 1158 raft_consensus.cc:738] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.533015 1158 consensus_queue.cc:260] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.534922 1055 raft_consensus.cc:2463] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:53.535636 1157 raft_consensus.cc:2238] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.535949 713 leader_election.cc:304] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.531693 1678 leader_election.cc:290] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.537477 1158 ts_tablet_manager.cc:1428] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20250109 20:52:53.537722 1158 tablet_bootstrap.cc:492] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.538770 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d773b9e15aff4f4fa339a5c6c2e70bd8" } modify_external_catalogs: true
I20250109 20:52:53.538930 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d773b9e15aff4f4fa339a5c6c2e70bd8" } modify_external_catalogs: true
I20250109 20:52:53.539075 1157 raft_consensus.cc:2267] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.539685 1158 tablet_bootstrap.cc:654] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.540836 757 tablet_service.cc:1514] Processing DeleteTablet for tablet dae574bf93d649099252e9e7b09b748c with delete_type TABLET_DATA_DELETED (replaced by tablet dc235614171f4a61b75492ddeb3f0fe6) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.541056 1796 raft_consensus.cc:2798] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:53.541173 1796 raft_consensus.cc:695] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:53.541419 1796 consensus_queue.cc:237] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.545429 1165 ts_tablet_manager.cc:1428] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.100s user 0.002s sys 0.000s
I20250109 20:52:53.549104 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6464281d788c4197b1e88847c593282c" } modify_external_catalogs: true
I20250109 20:52:53.549371 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6464281d788c4197b1e88847c593282c" } modify_external_catalogs: true
I20250109 20:52:53.551746 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a1bedae7fe314ca1bb17950aa1914398" } new_table_name: "default.Table_a4b39fd34a0e444cb84557a81907e39e" modify_external_catalogs: true
I20250109 20:52:53.552285 1796 consensus_queue.cc:1035] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:53.555297 1157 ts_tablet_manager.cc:1905] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.556896 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de7acfa7f941caa94bdbb888cf841e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:53.558230 1165 tablet_bootstrap.cc:492] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:53.560031 1165 tablet_bootstrap.cc:654] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.545869 1802 tablet.cc:1721] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:53.563021 1746 raft_consensus.cc:491] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.563190 1746 raft_consensus.cc:513] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
W20250109 20:52:53.564185 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9629fe30752f49478afad9b93dc235eb consensus state has no leader
I20250109 20:52:53.564378 1746 leader_election.cc:290] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:53.564695 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8654c154dab465d8991910698027c3f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.564826 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8654c154dab465d8991910698027c3f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:53.565321 989 leader_election.cc:343] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:53.565472 990 leader_election.cc:343] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.565618 990 leader_election.cc:304] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.565974 1746 raft_consensus.cc:2743] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:53.565994 448 catalog_manager.cc:5526] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:53.566479 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_bc43af02a045473cac4e0b21c2dfc318 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 1 got 0
W20250109 20:52:53.567435 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 274a58191d00475e8da9ff72dd33524f consensus state has no leader
I20250109 20:52:53.567893 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_44d9cfca7ff645bb8aad32946b74f286"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.568365 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44d9cfca7ff645bb8aad32946b74f286 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.568684 1176 tablet_bootstrap.cc:492] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.568826 1176 ts_tablet_manager.cc:1397] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.084s user 0.030s sys 0.019s
I20250109 20:52:53.569561 1176 raft_consensus.cc:357] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.569761 1176 raft_consensus.cc:383] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.569835 1176 raft_consensus.cc:738] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.570062 1176 consensus_queue.cc:260] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.570667 1176 ts_tablet_manager.cc:1428] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:53.571002 1176 tablet_bootstrap.cc:492] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.571206 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7f2a351186d44db99378b0fc3678356f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.571630 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f2a351186d44db99378b0fc3678356f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.572615 1176 tablet_bootstrap.cc:654] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.572697 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bdcb6780422541c89343e98e02938d0e
I20250109 20:52:53.573305 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_756f5f6ef85d468586a5fb77b0cc441d" } new_table_name: "default.Table_4560626d5c8547678952bb1e7da86ad4" modify_external_catalogs: true
I20250109 20:52:53.576117 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcb6780422541c89343e98e02938d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:53.576403 896 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcb6780422541c89343e98e02938d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:53.577586 1796 raft_consensus.cc:491] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.577739 1796 raft_consensus.cc:513] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.578361 1796 leader_election.cc:290] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:53.578456 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet bdcb6780422541c89343e98e02938d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdcb6780422541c89343e98e02938d0e already in progress: creating tablet
W20250109 20:52:53.578824 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e457baa02f4e48f8a627ffdb05267fc8 consensus state has no leader
I20250109 20:52:53.578889 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f494150713b4dde9399d53d57567983" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.579178 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_77c9fc51089f43c0a1c2db8a4d77573f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.579344 712 leader_election.cc:343] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:53.579702 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77c9fc51089f43c0a1c2db8a4d77573f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.580611 1157 ts_tablet_manager.cc:1918] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.580749 1157 log.cc:1198] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/540531463c184b9abf226d7487939e60
I20250109 20:52:53.581123 1157 ts_tablet_manager.cc:1939] T 540531463c184b9abf226d7487939e60 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:53.582175 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d7e19d2421e34bd98e4401879bd3277d
I20250109 20:52:53.583745 746 tablet_service.cc:1514] Processing DeleteTablet for tablet d7e19d2421e34bd98e4401879bd3277d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.583986 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet d7e19d2421e34bd98e4401879bd3277d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:53.584331 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d1722af0fbc4d2faa77942c00d9d3d6" } new_table_name: "default.Table_0cbc3b8995704eaf99466fd1d5d0e124" modify_external_catalogs: true
W20250109 20:52:53.584571 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d7e19d2421e34bd98e4401879bd3277d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e19d2421e34bd98e4401879bd3277d already in progress: creating tablet
I20250109 20:52:53.585222 896 tablet_service.cc:1514] Processing DeleteTablet for tablet d7e19d2421e34bd98e4401879bd3277d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:53.585492 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 540531463c184b9abf226d7487939e60 (table default.Table_642aeae240da4ea3a0fca064e8489134 [id=59a36af89f8f4df0ace607f3190236ba]) successfully deleted
I20250109 20:52:53.585829 1157 tablet_replica.cc:331] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:53.586021 1157 raft_consensus.cc:2238] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.586154 1157 raft_consensus.cc:2267] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.589345 1157 ts_tablet_manager.cc:1905] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.593868 757 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcb6780422541c89343e98e02938d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.594507 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f494150713b4dde9399d53d57567983" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:53.594744 1053 raft_consensus.cc:2463] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:53.599344 1678 raft_consensus.cc:491] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.599562 1678 raft_consensus.cc:513] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.600797 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b48900f57f47128e31b27275b1489c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.600844 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6dc6acbd34db412caf3281183b3c0f74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.601425 712 leader_election.cc:343] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:53.601469 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dc6acbd34db412caf3281183b3c0f74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.601740 1678 leader_election.cc:290] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.602450 1176 tablet_bootstrap.cc:492] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.604014 1157 ts_tablet_manager.cc:1918] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.604204 1157 log.cc:1198] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/b77c8d6adbf44375aff2782a83643255
I20250109 20:52:53.604262 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_84dfe41ea9994c1aa62b04e2ec77e4c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:53.604887 1157 ts_tablet_manager.cc:1939] T b77c8d6adbf44375aff2782a83643255 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:52:53.604940 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84dfe41ea9994c1aa62b04e2ec77e4c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.605794 1157 tablet_replica.cc:331] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:53.605890 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet b77c8d6adbf44375aff2782a83643255 (table default.Table_a5bb2f6d26e3423e93672a1954c238a7 [id=43c66cf1f4554c79b35bd2689050a945]) successfully deleted
I20250109 20:52:53.605965 1157 raft_consensus.cc:2238] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.606151 1157 raft_consensus.cc:2267] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.606902 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_50c2ff1060fd42b6a01b6b4f11cf3c14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.607299 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50c2ff1060fd42b6a01b6b4f11cf3c14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.607416 1176 ts_tablet_manager.cc:1397] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.005s
I20250109 20:52:53.607441 1746 raft_consensus.cc:491] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.607584 1746 raft_consensus.cc:513] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.608242 1746 leader_election.cc:290] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:53.608168 1176 raft_consensus.cc:357] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.608547 1176 raft_consensus.cc:383] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.608708 1176 raft_consensus.cc:738] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.608861 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773f2c6e6d7f415ab1218f2365c880a2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.608826 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "773f2c6e6d7f415ab1218f2365c880a2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:53.609103 1176 consensus_queue.cc:260] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:53.609398 989 leader_election.cc:343] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:53.609505 990 leader_election.cc:343] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.609611 989 leader_election.cc:304] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.609679 1176 ts_tablet_manager.cc:1428] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:53.609956 1746 raft_consensus.cc:2743] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:53.610435 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 825220629faa4d2481e0db9fa226c306 consensus state has no leader
I20250109 20:52:53.610716 1176 tablet_bootstrap.cc:492] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.610814 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8fdf081b7e1d4d55b7fc92577bfa1b3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.611287 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fdf081b7e1d4d55b7fc92577bfa1b3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.614027 1176 tablet_bootstrap.cc:654] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:53.614050 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ec1d30b67674facb8b92a2c1a408073 consensus state has no leader
I20250109 20:52:53.617462 1157 ts_tablet_manager.cc:1905] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.623168 1176 tablet_bootstrap.cc:492] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.623121 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b48900f57f47128e31b27275b1489c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:53.623284 1176 ts_tablet_manager.cc:1397] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20250109 20:52:53.623339 1053 raft_consensus.cc:2463] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:53.623749 713 leader_election.cc:304] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.624011 1678 raft_consensus.cc:2798] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:53.623973 1176 raft_consensus.cc:357] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.624114 1678 raft_consensus.cc:491] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:53.624243 1678 raft_consensus.cc:3054] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:53.624249 713 leader_election.cc:304] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.626677 1678 raft_consensus.cc:513] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.627405 1810 raft_consensus.cc:2798] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:53.627481 1810 raft_consensus.cc:491] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:53.627565 1810 raft_consensus.cc:3054] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:52:53.629475 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 04e46f4312844b6eb29d6718f4800f98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e46f4312844b6eb29d6718f4800f98 already in progress: creating tablet
W20250109 20:52:53.630213 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 189cd8079a9d4ca6b1703f4361d5ec5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 189cd8079a9d4ca6b1703f4361d5ec5d already in progress: creating tablet
I20250109 20:52:53.630214 1810 raft_consensus.cc:513] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:53.630829 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2ca83a49523b45be9ac7556b40746a83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ca83a49523b45be9ac7556b40746a83 already in progress: creating tablet
I20250109 20:52:53.631222 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b48900f57f47128e31b27275b1489c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:53.631366 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 687db3e734af47339c2546970a71a539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 687db3e734af47339c2546970a71a539 already in progress: creating tablet
I20250109 20:52:53.631749 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f494150713b4dde9399d53d57567983" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:53.631781 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ed00f7954143c9ba9bf65d8a8a68d2 already in progress: creating tablet
W20250109 20:52:53.632179 712 leader_election.cc:343] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:53.632239 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72cc0bbbb5584f14ba4d6c62211a12c5 already in progress: creating tablet
W20250109 20:52:53.632367 712 leader_election.cc:343] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.632611 1810 leader_election.cc:290] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.632880 1678 leader_election.cc:290] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.635896 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_487b0b221267468ba9603345f5030c8b [id=50837eb593734f66a03e29d8edbc63a4] alter complete (version 2)
I20250109 20:52:53.636829 746 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.637145 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acca65ecee94f8cbe3f2b89e744aa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:53.638283 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b48900f57f47128e31b27275b1489c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:53.638486 1053 raft_consensus.cc:3054] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:53.640550 1053 raft_consensus.cc:2463] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:53.641625 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f494150713b4dde9399d53d57567983" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:53.641837 1053 raft_consensus.cc:3054] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:53.645784 1053 raft_consensus.cc:2463] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:53.646561 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 75cdf5b737504d4e8f598cd2b265192e from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:53.649008 713 leader_election.cc:304] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.649287 1678 raft_consensus.cc:2798] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:53.649398 713 leader_election.cc:304] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.649439 1678 raft_consensus.cc:695] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:53.649823 1157 ts_tablet_manager.cc:1918] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.649753 1678 consensus_queue.cc:237] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.649916 1157 log.cc:1198] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/dae574bf93d649099252e9e7b09b748c
I20250109 20:52:53.650251 1157 ts_tablet_manager.cc:1939] T dae574bf93d649099252e9e7b09b748c P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:53.650771 1810 raft_consensus.cc:2798] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:53.650914 1810 raft_consensus.cc:695] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:53.651119 1810 consensus_queue.cc:237] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.651439 1157 tablet_replica.cc:331] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:53.651623 1157 raft_consensus.cc:2238] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.651762 1157 raft_consensus.cc:2267] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.652513 1157 ts_tablet_manager.cc:1905] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.652884 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet dae574bf93d649099252e9e7b09b748c (table default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]) successfully deleted
W20250109 20:52:53.653143 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d7e19d2421e34bd98e4401879bd3277d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e19d2421e34bd98e4401879bd3277d already in progress: creating tablet
W20250109 20:52:53.653393 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet bdcb6780422541c89343e98e02938d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdcb6780422541c89343e98e02938d0e already in progress: creating tablet
I20250109 20:52:53.659570 1176 raft_consensus.cc:383] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.659673 1176 raft_consensus.cc:738] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.659902 1176 consensus_queue.cc:260] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.660400 1176 ts_tablet_manager.cc:1428] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.037s user 0.012s sys 0.001s
I20250109 20:52:53.661149 1157 ts_tablet_manager.cc:1918] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.661336 1157 log.cc:1198] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/fbde2eecdaf4470dbc74b0c59f31643c
I20250109 20:52:53.661757 1157 ts_tablet_manager.cc:1939] T fbde2eecdaf4470dbc74b0c59f31643c P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:53.662884 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet fbde2eecdaf4470dbc74b0c59f31643c (table default.Table_56c8a1eb8409437c856c4af9ece36c61 [id=fd0bcd24e74342f299f6b060b684af79]) successfully deleted
I20250109 20:52:53.663304 1157 tablet_replica.cc:331] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:53.663540 1157 raft_consensus.cc:2238] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.663766 1157 raft_consensus.cc:2267] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.664503 1157 ts_tablet_manager.cc:1905] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.670190 1771 consensus_queue.cc:1035] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:53.671046 1157 ts_tablet_manager.cc:1918] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.671236 1157 log.cc:1198] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6acca65ecee94f8cbe3f2b89e744aa6c
I20250109 20:52:53.671722 1157 ts_tablet_manager.cc:1939] T 6acca65ecee94f8cbe3f2b89e744aa6c P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:53.672883 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6acca65ecee94f8cbe3f2b89e744aa6c (table default.Table_4792b8689a12402f9946e87ad47dcbbb [id=ec2d9d93307747a7b752255f2008500e]) successfully deleted
I20250109 20:52:53.674017 1771 consensus_queue.cc:1035] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:53.678933 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 712acab92e0f44a3a585d7d871498aa3 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:53.680276 1158 tablet_bootstrap.cc:492] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.680557 1158 ts_tablet_manager.cc:1397] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.143s user 0.019s sys 0.031s
I20250109 20:52:53.681389 1158 raft_consensus.cc:357] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.681627 1158 raft_consensus.cc:383] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.681761 1158 raft_consensus.cc:738] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:53.681806 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 80da78917f0b4931ab1705371c1fe5dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80da78917f0b4931ab1705371c1fe5dd already in progress: creating tablet
I20250109 20:52:53.682009 1158 consensus_queue.cc:260] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.682471 1158 ts_tablet_manager.cc:1428] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250109 20:52:53.682469 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2797e1df3d12476f89fb5155523c3935 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2797e1df3d12476f89fb5155523c3935 already in progress: creating tablet
I20250109 20:52:53.682703 1158 tablet_bootstrap.cc:492] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:53.683068 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 678ab724db1a42e2b23bbe6cd7652db7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 678ab724db1a42e2b23bbe6cd7652db7 already in progress: creating tablet
W20250109 20:52:53.683676 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 418e1bd414914b21a233c3099e2057da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 418e1bd414914b21a233c3099e2057da already in progress: creating tablet
W20250109 20:52:53.684478 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e0f67c4294f8499496db4e82db8c3bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f67c4294f8499496db4e82db8c3bf0 already in progress: creating tablet
I20250109 20:52:53.684863 1158 tablet_bootstrap.cc:654] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.686425 1165 tablet_bootstrap.cc:492] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:53.686939 1176 tablet_bootstrap.cc:492] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.688179 1176 tablet_bootstrap.cc:654] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.694885 1165 ts_tablet_manager.cc:1397] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.137s user 0.001s sys 0.002s
I20250109 20:52:53.695662 1165 raft_consensus.cc:357] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.696835 1165 raft_consensus.cc:383] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.697010 1165 raft_consensus.cc:738] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.694453 1160 tablet_replica.cc:331] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:53.698099 1160 raft_consensus.cc:2238] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.698364 1160 raft_consensus.cc:2267] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.703727 1176 tablet_bootstrap.cc:492] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.703907 1176 ts_tablet_manager.cc:1397] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20250109 20:52:53.704568 1176 raft_consensus.cc:357] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.704833 1176 raft_consensus.cc:383] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.704931 1176 raft_consensus.cc:738] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.705178 1176 consensus_queue.cc:260] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.705698 1176 ts_tablet_manager.cc:1428] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:53.705925 1176 tablet_bootstrap.cc:492] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.707310 1176 tablet_bootstrap.cc:654] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.709247 1158 tablet_bootstrap.cc:492] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.709396 1158 ts_tablet_manager.cc:1397] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20250109 20:52:53.710529 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b47656fe1ae4c9ba4138bcd37c0e6db from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:53.711388 1160 ts_tablet_manager.cc:1905] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.711913 1158 raft_consensus.cc:357] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.712175 1158 raft_consensus.cc:383] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.712283 1158 raft_consensus.cc:738] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.712530 1158 consensus_queue.cc:260] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.713045 1158 ts_tablet_manager.cc:1428] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:53.713374 1158 tablet_bootstrap.cc:492] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.713481 1176 tablet_bootstrap.cc:492] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.713631 1176 ts_tablet_manager.cc:1397] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250109 20:52:53.714239 1176 raft_consensus.cc:357] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.714484 1176 raft_consensus.cc:383] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.714586 1176 raft_consensus.cc:738] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.714844 1176 consensus_queue.cc:260] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.715296 1158 tablet_bootstrap.cc:654] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.715725 1176 ts_tablet_manager.cc:1428] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:53.715988 1176 tablet_bootstrap.cc:492] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.716465 1165 consensus_queue.cc:260] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.717440 1176 tablet_bootstrap.cc:654] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.738894 1832 tablet.cc:1721] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:53.741640 1826 raft_consensus.cc:491] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.741886 1826 raft_consensus.cc:513] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.742671 1826 leader_election.cc:290] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.744422 1176 tablet_bootstrap.cc:492] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.744602 1176 ts_tablet_manager.cc:1397] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.005s
I20250109 20:52:53.745707 1802 tablet.cc:1721] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:53.745735 1176 raft_consensus.cc:357] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.746112 1176 raft_consensus.cc:383] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.746235 1176 raft_consensus.cc:738] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.746735 1803 tablet.cc:1721] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:53.747987 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 3000b819a4cc4960909b98febd400631 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c65489a602141239c99e8e8dae22ab4) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:53.748605 854 leader_election.cc:343] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 3000b819a4cc4960909b98febd400631
I20250109 20:52:53.750746 1165 ts_tablet_manager.cc:1428] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20250109 20:52:53.751266 1823 consensus_queue.cc:1035] T 2ab7ca098694464faec002cec05558ac P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:53.757175 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_65bd13ddff1b46929045b98b3c549a18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:53.757901 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_881894033dec46b3862c1185df595c40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.758957 854 leader_election.cc:343] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 3000b819a4cc4960909b98febd400631
I20250109 20:52:53.759132 854 leader_election.cc:304] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:53.759603 1823 raft_consensus.cc:2743] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:53.762696 1176 consensus_queue.cc:260] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.766494 1176 ts_tablet_manager.cc:1428] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20250109 20:52:53.766835 1176 tablet_bootstrap.cc:492] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.768474 1176 tablet_bootstrap.cc:654] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.772141 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dd4f794be11b44ad971a871fdbea1094" } new_table_name: "default.Table_d2d8f346b1d2494cb588d606237080a0" modify_external_catalogs: true
I20250109 20:52:53.774245 1826 consensus_queue.cc:1035] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:52:53.779601 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74f4bdda16554d9d81de5ca5c5205f4c consensus state has no leader
I20250109 20:52:53.780809 1158 tablet_bootstrap.cc:492] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.781201 1158 ts_tablet_manager.cc:1397] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.022s
I20250109 20:52:53.781997 1158 raft_consensus.cc:357] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.782253 1158 raft_consensus.cc:383] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.782253 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e1503bbd879343928760e2b8a1192c3a" } new_table_name: "default.Table_71c13b2579c040b69ad859670b25a737" modify_external_catalogs: true
I20250109 20:52:53.782377 1158 raft_consensus.cc:738] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.782783 1158 consensus_queue.cc:260] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.783424 1158 ts_tablet_manager.cc:1428] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:53.785789 1158 tablet_bootstrap.cc:492] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:53.789161 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a1d5b0210134ce1a33b5e7603998593 consensus state has no leader
I20250109 20:52:53.789908 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d72a28b8d2b94712805f7299f0ccccf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:53.791332 1158 tablet_bootstrap.cc:654] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.802487 1158 tablet_bootstrap.cc:492] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.802536 1165 tablet_bootstrap.cc:492] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:53.802873 1158 ts_tablet_manager.cc:1397] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20250109 20:52:53.805073 1165 tablet_bootstrap.cc:654] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.804911 1158 raft_consensus.cc:357] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.805584 1158 raft_consensus.cc:383] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.805724 1158 raft_consensus.cc:738] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.806121 1158 consensus_queue.cc:260] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.806912 1158 ts_tablet_manager.cc:1428] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:53.807754 1158 tablet_bootstrap.cc:492] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.810374 1158 tablet_bootstrap.cc:654] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.811041 1833 tablet.cc:1721] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:53.815534 1160 ts_tablet_manager.cc:1918] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.816313 1160 log.cc:1198] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/41de7acfa7f941caa94bdbb888cf841e
I20250109 20:52:53.817116 1160 ts_tablet_manager.cc:1939] T 41de7acfa7f941caa94bdbb888cf841e P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:53.817912 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 41de7acfa7f941caa94bdbb888cf841e (table default.Table_65cb0c2ed23b450ca192c6f61becbc51 [id=2504d7b1b64f477a8b7763d5d0a9ec6e]) successfully deleted
W20250109 20:52:53.830570 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65bd13ddff1b46929045b98b3c549a18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.830896 1176 tablet_bootstrap.cc:492] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.831018 1176 ts_tablet_manager.cc:1397] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.009s
I20250109 20:52:53.831635 1176 raft_consensus.cc:357] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.831789 1176 raft_consensus.cc:383] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.831831 1176 raft_consensus.cc:738] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.832031 1176 consensus_queue.cc:260] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.832547 1176 ts_tablet_manager.cc:1428] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:52:53.832790 1176 tablet_bootstrap.cc:492] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:53.830569 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d72a28b8d2b94712805f7299f0ccccf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.834547 1806 raft_consensus.cc:491] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.834810 1176 tablet_bootstrap.cc:654] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.834685 1806 raft_consensus.cc:513] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.835335 1806 leader_election.cc:290] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:53.836227 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e6e2b3b9a4f9f9d08361ba1cbb5ba" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:53.836843 990 leader_election.cc:343] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.837146 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e6e2b3b9a4f9f9d08361ba1cbb5ba" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:53.837893 989 leader_election.cc:343] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.838002 989 leader_election.cc:304] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.838338 1806 raft_consensus.cc:2743] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:53.845377 1176 tablet_bootstrap.cc:492] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.845580 1176 ts_tablet_manager.cc:1397] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250109 20:52:53.846432 1176 raft_consensus.cc:357] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.846657 1176 raft_consensus.cc:383] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.846740 1176 raft_consensus.cc:738] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.847752 1176 consensus_queue.cc:260] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.848223 1826 consensus_queue.cc:1035] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:53.848438 1176 ts_tablet_manager.cc:1428] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 20:52:53.849434 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82] alter complete (version 1)
I20250109 20:52:53.849906 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1cf2c16f6ff048719047b27308f7cf99" } modify_external_catalogs: true
I20250109 20:52:53.850055 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1cf2c16f6ff048719047b27308f7cf99" } modify_external_catalogs: true
I20250109 20:52:53.850330 1176 tablet_bootstrap.cc:492] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.852120 1176 tablet_bootstrap.cc:654] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:53.830606 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_881894033dec46b3862c1185df595c40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.864081 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 728e9b6101e3447fb3c24b3917aa4872 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:53.864722 1176 tablet_bootstrap.cc:492] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.864909 1176 ts_tablet_manager.cc:1397] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20250109 20:52:53.865303 1806 raft_consensus.cc:491] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.865470 1806 raft_consensus.cc:513] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.865897 1176 raft_consensus.cc:357] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.866114 1176 raft_consensus.cc:383] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.866663 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5d462230734767bb010f520c86c5ab" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.866748 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5d462230734767bb010f520c86c5ab" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:53.866202 1176 raft_consensus.cc:738] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.866228 1806 leader_election.cc:290] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:53.867172 989 leader_election.cc:343] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:53.867744 990 leader_election.cc:343] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.867838 990 leader_election.cc:304] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.867853 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_278a8479a4ac4de89f6895774e7eeeaa" } new_table_name: "default.Table_14c64344192b45778141de193a1c433d" modify_external_catalogs: true
I20250109 20:52:53.868203 1806 raft_consensus.cc:2743] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:53.867569 1176 consensus_queue.cc:260] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.868937 1176 ts_tablet_manager.cc:1428] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250109 20:52:53.869215 1176 tablet_bootstrap.cc:492] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.870920 1176 tablet_bootstrap.cc:654] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:53.873472 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c3af2a00ab04b41915cffc3eb9206bc consensus state has no leader
I20250109 20:52:53.874099 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e4be594b7e46489d83d7c7f8bbbed681" } modify_external_catalogs: true
I20250109 20:52:53.874374 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e4be594b7e46489d83d7c7f8bbbed681" } modify_external_catalogs: true
I20250109 20:52:53.875303 1158 tablet_bootstrap.cc:492] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.875449 1158 ts_tablet_manager.cc:1397] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.068s user 0.020s sys 0.025s
I20250109 20:52:53.876226 1158 raft_consensus.cc:357] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.876415 1158 raft_consensus.cc:383] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.876480 1158 raft_consensus.cc:738] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.876976 1158 consensus_queue.cc:260] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.877539 1158 ts_tablet_manager.cc:1428] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:53.877880 1158 tablet_bootstrap.cc:492] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:53.878713 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu pid 1296: Timed out: CheckLeaks RPC to 127.0.104.60:41583 timed out after 1.000s (SENT)
I20250109 20:52:53.878865 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu with pid 1296
I20250109 20:52:53.881528 1158 tablet_bootstrap.cc:654] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.882903 1176 tablet_bootstrap.cc:492] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.883077 1176 ts_tablet_manager.cc:1397] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20250109 20:52:53.883797 1176 raft_consensus.cc:357] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.885932 1176 raft_consensus.cc:383] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.886135 1176 raft_consensus.cc:738] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.886484 1165 tablet_bootstrap.cc:492] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:53.886685 1165 ts_tablet_manager.cc:1397] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.084s user 0.001s sys 0.003s
I20250109 20:52:53.886426 1176 consensus_queue.cc:260] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:53.887105 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet bdcb6780422541c89343e98e02938d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdcb6780422541c89343e98e02938d0e already in progress: creating tablet
I20250109 20:52:53.887576 1176 ts_tablet_manager.cc:1428] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:53.887622 1165 raft_consensus.cc:357] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.887878 1165 raft_consensus.cc:383] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.887987 1165 raft_consensus.cc:738] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.888199 1176 tablet_bootstrap.cc:492] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:53.888298 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d7e19d2421e34bd98e4401879bd3277d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e19d2421e34bd98e4401879bd3277d already in progress: creating tablet
I20250109 20:52:53.888237 1165 consensus_queue.cc:260] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.888844 1165 ts_tablet_manager.cc:1428] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:53.890780 1176 tablet_bootstrap.cc:654] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.893260 1771 consensus_queue.cc:1035] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:53.895355 1158 tablet_bootstrap.cc:492] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.895566 1158 ts_tablet_manager.cc:1397] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.002s
I20250109 20:52:53.896389 1158 raft_consensus.cc:357] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.896701 1158 raft_consensus.cc:383] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.896839 1158 raft_consensus.cc:738] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.897120 1158 consensus_queue.cc:260] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.897699 1158 ts_tablet_manager.cc:1428] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:53.898063 1158 tablet_bootstrap.cc:492] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.899849 1158 tablet_bootstrap.cc:654] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.900734 1165 tablet_bootstrap.cc:492] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:53.903079 1160 tablet_replica.cc:331] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:53.908193 1160 raft_consensus.cc:2238] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:53.908959 1160 raft_consensus.cc:2267] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:53.910002 1160 ts_tablet_manager.cc:1905] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:53.910547 1806 raft_consensus.cc:491] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.910650 1806 raft_consensus.cc:513] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.911159 1806 leader_election.cc:290] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:53.911675 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e615ce8e124cfe9432778b6b58e117" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:53.911911 1158 tablet_bootstrap.cc:492] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
W20250109 20:52:53.912155 990 leader_election.cc:343] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.912164 1158 ts_tablet_manager.cc:1397] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20250109 20:52:53.913251 1158 raft_consensus.cc:357] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.913489 1158 raft_consensus.cc:383] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.913596 1158 raft_consensus.cc:738] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.910337 1165 tablet_bootstrap.cc:654] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.912653 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e615ce8e124cfe9432778b6b58e117" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.913930 1158 consensus_queue.cc:260] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
W20250109 20:52:53.914124 989 leader_election.cc:343] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.914276 989 leader_election.cc:304] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.914758 1806 raft_consensus.cc:2743] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:53.914775 1158 ts_tablet_manager.cc:1428] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:53.915133 1158 tablet_bootstrap.cc:492] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.917030 1158 tablet_bootstrap.cc:654] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.925035 1158 tablet_bootstrap.cc:492] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.925279 1158 ts_tablet_manager.cc:1397] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20250109 20:52:53.925976 1158 raft_consensus.cc:357] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.926234 1158 raft_consensus.cc:383] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.926344 1158 raft_consensus.cc:738] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.926605 1158 consensus_queue.cc:260] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.926689 1806 raft_consensus.cc:491] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.926808 1806 raft_consensus.cc:513] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.927165 1158 ts_tablet_manager.cc:1428] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:53.927343 1806 leader_election.cc:290] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:53.927466 1158 tablet_bootstrap.cc:492] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.928216 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2608f48f5950466e9ab2d12ee87503cb" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:53.928792 989 leader_election.cc:343] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.929088 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2608f48f5950466e9ab2d12ee87503cb" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:53.929334 1158 tablet_bootstrap.cc:654] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:53.929535 990 leader_election.cc:343] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.929626 990 leader_election.cc:304] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.929991 1806 raft_consensus.cc:2743] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:53.937258 1158 tablet_bootstrap.cc:492] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.937438 1158 ts_tablet_manager.cc:1397] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.001s
I20250109 20:52:53.937861 1176 tablet_bootstrap.cc:492] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.938009 1176 ts_tablet_manager.cc:1397] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.008s
I20250109 20:52:53.938243 1158 raft_consensus.cc:357] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.938477 1158 raft_consensus.cc:383] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.938593 1158 raft_consensus.cc:738] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.938688 1176 raft_consensus.cc:357] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.938854 1176 raft_consensus.cc:383] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.938912 1176 raft_consensus.cc:738] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.938937 1158 consensus_queue.cc:260] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.939091 1176 consensus_queue.cc:260] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.939525 1176 ts_tablet_manager.cc:1428] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:52:53.939555 1158 ts_tablet_manager.cc:1428] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:53.939786 1176 tablet_bootstrap.cc:492] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.939918 1158 tablet_bootstrap.cc:492] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.941411 1176 tablet_bootstrap.cc:654] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.941766 1158 tablet_bootstrap.cc:654] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.942631 1165 tablet_bootstrap.cc:492] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:53.942783 1165 ts_tablet_manager.cc:1397] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.002s
I20250109 20:52:53.943473 1165 raft_consensus.cc:357] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.943645 1165 raft_consensus.cc:383] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.943722 1165 raft_consensus.cc:738] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.943936 1165 consensus_queue.cc:260] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.944605 1165 ts_tablet_manager.cc:1428] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:53.944877 1165 tablet_bootstrap.cc:492] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:53.946522 1165 tablet_bootstrap.cc:654] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.947995 1160 ts_tablet_manager.cc:1918] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:53.948119 1160 log.cc:1198] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/3000b819a4cc4960909b98febd400631
I20250109 20:52:53.948459 1160 ts_tablet_manager.cc:1939] T 3000b819a4cc4960909b98febd400631 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:53.949103 1158 tablet_bootstrap.cc:492] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.949152 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 3000b819a4cc4960909b98febd400631 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]) successfully deleted
I20250109 20:52:53.949311 1158 ts_tablet_manager.cc:1397] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20250109 20:52:53.949923 1158 raft_consensus.cc:357] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.950150 1158 raft_consensus.cc:383] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.950212 1158 raft_consensus.cc:738] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.950445 1158 consensus_queue.cc:260] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.951056 1158 ts_tablet_manager.cc:1428] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:53.951357 1158 tablet_bootstrap.cc:492] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.954334 1158 tablet_bootstrap.cc:654] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.956027 1176 tablet_bootstrap.cc:492] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.956203 1176 ts_tablet_manager.cc:1397] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20250109 20:52:53.957103 1176 raft_consensus.cc:357] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.957343 1176 raft_consensus.cc:383] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.957424 1176 raft_consensus.cc:738] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.957696 1176 consensus_queue.cc:260] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.958276 1176 ts_tablet_manager.cc:1428] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:53.958536 1176 tablet_bootstrap.cc:492] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.958895 1158 tablet_bootstrap.cc:492] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.959064 1158 ts_tablet_manager.cc:1397] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20250109 20:52:53.959789 1158 raft_consensus.cc:357] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.959908 1176 tablet_bootstrap.cc:654] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.960029 1158 raft_consensus.cc:383] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.960161 1158 raft_consensus.cc:738] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.960413 1158 consensus_queue.cc:260] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:53.961024 1158 ts_tablet_manager.cc:1428] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:53.961320 1158 tablet_bootstrap.cc:492] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.962425 1806 raft_consensus.cc:491] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.962551 1806 raft_consensus.cc:513] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.963004 1158 tablet_bootstrap.cc:654] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.963151 1806 leader_election.cc:290] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:53.963750 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7307ebdb0c794451a6d91d23d69fddc6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:53.964295 990 leader_election.cc:343] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.965019 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7307ebdb0c794451a6d91d23d69fddc6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.965332 1158 tablet_bootstrap.cc:492] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:53.965529 1158 ts_tablet_manager.cc:1397] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
W20250109 20:52:53.965626 989 leader_election.cc:343] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.965726 989 leader_election.cc:304] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:53.966192 1806 raft_consensus.cc:2743] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:53.966600 1158 raft_consensus.cc:357] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.966876 1158 raft_consensus.cc:383] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.966986 1158 raft_consensus.cc:738] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.967407 1158 consensus_queue.cc:260] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.967996 1158 ts_tablet_manager.cc:1428] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:53.968302 1158 tablet_bootstrap.cc:492] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:53.968569 1810 raft_consensus.cc:491] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.968775 1810 raft_consensus.cc:513] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.969445 1810 leader_election.cc:290] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.970259 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64207928ef141118e7fc8fe7bb2379b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:53.970702 713 leader_election.cc:343] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: e64207928ef141118e7fc8fe7bb2379b
W20250109 20:52:53.970830 712 leader_election.cc:343] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:53.970955 712 leader_election.cc:304] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:53.971297 1678 raft_consensus.cc:491] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:53.971405 1678 raft_consensus.cc:513] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.971710 1176 tablet_bootstrap.cc:492] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:53.971812 1176 ts_tablet_manager.cc:1397] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250109 20:52:53.971911 1678 leader_election.cc:290] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.972038 1678 raft_consensus.cc:2743] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:53.972172 853 connection.cc:537] client connection to 127.0.104.60:41583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250109 20:52:53.972353 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.60:41583 (error 108) [suppressed 8 similar messages]
I20250109 20:52:53.972445 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8e7081cac440389bec5dfac2391beb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:53.972608 1055 raft_consensus.cc:2463] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
W20250109 20:52:53.972920 989 connection.cc:537] client connection to 127.0.104.60:41583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250109 20:52:53.972934 713 leader_election.cc:304] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:53.973155 1176 raft_consensus.cc:357] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.973317 1176 raft_consensus.cc:383] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:53.973361 1176 raft_consensus.cc:738] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:53.970964 1158 tablet_bootstrap.cc:654] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.973526 1176 consensus_queue.cc:260] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:53.973924 712 connection.cc:537] client connection to 127.0.104.60:41583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250109 20:52:53.973975 1176 ts_tablet_manager.cc:1428] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:53.974195 1176 tablet_bootstrap.cc:492] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:53.975032 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8e7081cac440389bec5dfac2391beb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:53.975927 1678 raft_consensus.cc:2798] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:53.976027 1678 raft_consensus.cc:491] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:53.976121 1678 raft_consensus.cc:3054] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:53.976548 1176 tablet_bootstrap.cc:654] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:53.978720 1678 raft_consensus.cc:513] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:53.979269 1678 leader_election.cc:290] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:53.990478 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 2608f48f5950466e9ab2d12ee87503cb (table default.Table_8a7392fff9c0401fb5c701ee92cf9c41 [id=17d12fac83d6463fb91fab6729a08139]). Expected version 1 got 0
I20250109 20:52:53.993049 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2768784b71f9461bbe4081deaaae9588"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:53.993659 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2768784b71f9461bbe4081deaaae9588 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:53.994658 448 catalog_manager.cc:5526] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:53.995184 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
I20250109 20:52:53.995491 448 catalog_manager.cc:5526] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:54.002643 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 246fd9ac96f54d008e8203e4580e7ef3
I20250109 20:52:54.003086 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2be32f590fd844b2bbff91488a5390fc
I20250109 20:52:54.006245 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.006525 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2be32f590fd844b2bbff91488a5390fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.007359 1810 raft_consensus.cc:491] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.007514 1810 raft_consensus.cc:513] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
W20250109 20:52:54.007603 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 246fd9ac96f54d008e8203e4580e7ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 246fd9ac96f54d008e8203e4580e7ef3 already in progress: creating tablet
W20250109 20:52:54.007882 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2be32f590fd844b2bbff91488a5390fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2be32f590fd844b2bbff91488a5390fc already in progress: creating tablet
I20250109 20:52:54.008081 1810 leader_election.cc:290] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.008527 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2be32f590fd844b2bbff91488a5390fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.010339 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8e7081cac440389bec5dfac2391beb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:54.011013 712 leader_election.cc:343] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.011626 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:54.012454 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2be32f590fd844b2bbff91488a5390fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2be32f590fd844b2bbff91488a5390fc already in progress: creating tablet
W20250109 20:52:54.012789 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 246fd9ac96f54d008e8203e4580e7ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 246fd9ac96f54d008e8203e4580e7ef3 already in progress: creating tablet
W20250109 20:52:54.014966 712 leader_election.cc:343] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.015554 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 65570e42a9bd45bcba0a82abd12a9356
I20250109 20:52:54.015645 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d4cad7594346dfafdee8e4c49d74e6" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.016258 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0309fd73acdf416d94c30a93bc3e1cfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.016369 712 leader_election.cc:343] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:54.016775 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0309fd73acdf416d94c30a93bc3e1cfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.017670 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8e7081cac440389bec5dfac2391beb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:54.017850 1055 raft_consensus.cc:3054] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.018167 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5a36dfb8096e4cf79407414ba122d0e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.018576 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a36dfb8096e4cf79407414ba122d0e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.020057 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 65570e42a9bd45bcba0a82abd12a9356 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.020313 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 65570e42a9bd45bcba0a82abd12a9356 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.021008 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_17129d3d3fd24e9386fd3228f1e15342" } new_table_name: "default.Table_5c79be4802d64c3f8e53093dba415e6e" modify_external_catalogs: true
I20250109 20:52:54.021386 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:54.022697 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f32d9ff86dad43228e89600903a000f8" } new_table_name: "default.Table_ad073c54c6aa42579dde0692025efcef" modify_external_catalogs: true
W20250109 20:52:54.024472 713 leader_election.cc:343] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 46d4cad7594346dfafdee8e4c49d74e6
I20250109 20:52:54.024586 713 leader_election.cc:304] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:54.024893 1810 raft_consensus.cc:2743] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:54.026643 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 65570e42a9bd45bcba0a82abd12a9356 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65570e42a9bd45bcba0a82abd12a9356 already in progress: creating tablet
W20250109 20:52:54.026952 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 65570e42a9bd45bcba0a82abd12a9356 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65570e42a9bd45bcba0a82abd12a9356 already in progress: creating tablet
I20250109 20:52:54.020328 1055 raft_consensus.cc:2463] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:54.028210 713 leader_election.cc:304] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.028548 1810 raft_consensus.cc:2798] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:54.028784 1810 raft_consensus.cc:695] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:54.029036 1810 consensus_queue.cc:237] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.030524 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 728e9b6101e3447fb3c24b3917aa4872
I20250109 20:52:54.031070 453 catalog_manager.cc:6596] ReplaceTablet: tablet 728e9b6101e3447fb3c24b3917aa4872 deleted and replaced by tablet d79437d800f44c5a8222eea7dc2cd4e3
I20250109 20:52:54.033589 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 with delete_type TABLET_DATA_DELETED (replaced by tablet d79437d800f44c5a8222eea7dc2cd4e3) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.033932 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 with delete_type TABLET_DATA_DELETED (replaced by tablet d79437d800f44c5a8222eea7dc2cd4e3) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:54.035043 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 728e9b6101e3447fb3c24b3917aa4872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728e9b6101e3447fb3c24b3917aa4872 already in progress: creating tablet
I20250109 20:52:54.035862 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 712acab92e0f44a3a585d7d871498aa3
W20250109 20:52:54.035553 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 728e9b6101e3447fb3c24b3917aa4872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728e9b6101e3447fb3c24b3917aa4872 already in progress: creating tablet
I20250109 20:52:54.037516 451 catalog_manager.cc:6596] ReplaceTablet: tablet 712acab92e0f44a3a585d7d871498aa3 deleted and replaced by tablet 6e020eeb03e041ed977779a07de5bb80
I20250109 20:52:54.038018 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a3861f342059441c9d03405107eb97c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.038560 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3861f342059441c9d03405107eb97c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.040400 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 712acab92e0f44a3a585d7d871498aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e020eeb03e041ed977779a07de5bb80) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.041081 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 712acab92e0f44a3a585d7d871498aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e020eeb03e041ed977779a07de5bb80) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:54.041796 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 712acab92e0f44a3a585d7d871498aa3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 712acab92e0f44a3a585d7d871498aa3 already in progress: creating tablet
W20250109 20:52:54.041992 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 712acab92e0f44a3a585d7d871498aa3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 712acab92e0f44a3a585d7d871498aa3 already in progress: creating tablet
I20250109 20:52:54.050225 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 75cdf5b737504d4e8f598cd2b265192e
W20250109 20:52:54.050305 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c9a4ad54fef41c3941fd9a02be5a418 has no consensus state
I20250109 20:52:54.050621 446 catalog_manager.cc:6596] ReplaceTablet: tablet 75cdf5b737504d4e8f598cd2b265192e deleted and replaced by tablet 1591374d521a47ebb405a812fe656b19
I20250109 20:52:54.050969 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0a8d43084a0e45b48d75867cfcbe83e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:54.051503 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cdf5b737504d4e8f598cd2b265192e with delete_type TABLET_DATA_DELETED (replaced by tablet 1591374d521a47ebb405a812fe656b19) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:54.051621 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a8d43084a0e45b48d75867cfcbe83e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.052235 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cdf5b737504d4e8f598cd2b265192e with delete_type TABLET_DATA_DELETED (replaced by tablet 1591374d521a47ebb405a812fe656b19) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.053009 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_684bc1fac0764a80acc3c9be2f5df7ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:54.053315 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_98aa779b222a4fa1bb03d9c1d84dc2bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.053547 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_684bc1fac0764a80acc3c9be2f5df7ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:54.053750 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98aa779b222a4fa1bb03d9c1d84dc2bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:54.054994 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 75cdf5b737504d4e8f598cd2b265192e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75cdf5b737504d4e8f598cd2b265192e already in progress: creating tablet
W20250109 20:52:54.055266 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 75cdf5b737504d4e8f598cd2b265192e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75cdf5b737504d4e8f598cd2b265192e already in progress: creating tablet
I20250109 20:52:54.056218 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f68bc0a564c4210bb29172a8759ebf3 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:54.058162 1176 tablet_bootstrap.cc:492] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.058383 1176 ts_tablet_manager.cc:1397] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.084s user 0.023s sys 0.016s
I20250109 20:52:54.062511 1176 raft_consensus.cc:357] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.062762 1176 raft_consensus.cc:383] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.062891 1176 raft_consensus.cc:738] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.063421 1176 consensus_queue.cc:260] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.067423 1176 ts_tablet_manager.cc:1428] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250109 20:52:54.069867 1176 tablet_bootstrap.cc:492] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:54.070688 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d845bac2539e478083bf7fe6cbc0c717 consensus state has no leader
I20250109 20:52:54.070938 746 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9a4ad54fef41c3941fd9a02be5a418 (DEFAULT_TABLE table=default.Table_ad073c54c6aa42579dde0692025efcef [id=560a2c7adf7e45bfa286b1d6ad6e3083]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.071410 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9a4ad54fef41c3941fd9a02be5a418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.072067 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:54.072764 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:52:54.072990 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:54.077901 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_cbde6546bcd3407ebd6470f02d881bdd [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 4 got 0
I20250109 20:52:54.078091 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:54.078465 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:54.079171 1176 tablet_bootstrap.cc:654] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.079332 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c6662f9bad8425db77969d4150d9c38" } new_table_name: "default.Table_017eb9f961124f788b1a150416f6ccb3" modify_external_catalogs: true
I20250109 20:52:54.081689 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d2ad301c11943b69b59be18d6775513 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:54.082954 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_759654c8879645ee84625859a22e72b2" } new_table_name: "default.Table_291aa89efe4c406cbebcd12163ceae90" modify_external_catalogs: true
I20250109 20:52:54.085417 757 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcadec0c3184feaad4a883e5588f310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.085696 744 tablet_service.cc:1514] Processing DeleteTablet for tablet e64207928ef141118e7fc8fe7bb2379b with delete_type TABLET_DATA_DELETED (replaced by tablet 2797e1df3d12476f89fb5155523c3935) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.085875 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.086055 754 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.086246 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 402280f75e51463a865c0fb521d0fd8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.086452 751 tablet_service.cc:1514] Processing DeleteTablet for tablet cba68160be6145ddbbd2019c9b5f0076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.086653 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8e7081cac440389bec5dfac2391beb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.086824 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 with delete_type TABLET_DATA_DELETED (replaced by tablet ed5ea5d588834f1b8b2b057ec324aadc) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.083526 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3a97d64313b6410788dafa912c301267"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.087226 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a97d64313b6410788dafa912c301267 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.088163 1157 tablet_replica.cc:331] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.088196 1806 raft_consensus.cc:491] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.088353 1157 raft_consensus.cc:2238] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.088351 1806 raft_consensus.cc:513] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.088503 1157 raft_consensus.cc:2267] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.089073 1806 leader_election.cc:290] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.089977 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2b022ad9274c0e9b67560389d8dbed" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:54.090560 989 leader_election.cc:343] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.090638 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b0450d88e3e4ccaae4e5157ce5c80c1) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.090827 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet b221e048f6f6463e84e630386c304a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.090893 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.091018 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8e7081cac440389bec5dfac2391beb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.091135 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.091193 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.091343 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet deed945f2ecf4290ad002662c3f66030 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.091289 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2b022ad9274c0e9b67560389d8dbed" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.091506 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.090827 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5d462230734767bb010f520c86c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.091037 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757b98100804c688f67e7111db2b622 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f8419d8cc7e475ea43dd263cae87aca) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.091780 1166 tablet_replica.cc:331] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:54.091831 990 leader_election.cc:343] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.091938 1166 raft_consensus.cc:2238] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.092072 1166 raft_consensus.cc:2267] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.092281 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a455033b674e5a94fc37d0275eb313 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.091948 990 leader_election.cc:304] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.092451 1157 ts_tablet_manager.cc:1905] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.092767 1806 raft_consensus.cc:2743] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.092731 1166 ts_tablet_manager.cc:1905] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.093335 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0915efe95c4141fa8a554a6b8fb28fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.093923 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0915efe95c4141fa8a554a6b8fb28fea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.093966 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
I20250109 20:52:54.094727 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7420a91ef4554611bfbb139ea53bd075"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:54.096318 746 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 with delete_type TABLET_DATA_DELETED (replaced by tablet 6619b3dce3ed4325a4871c3d40b42c9d) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.098908 1157 ts_tablet_manager.cc:1918] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.099094 1157 log.cc:1198] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/fbcadec0c3184feaad4a883e5588f310
I20250109 20:52:54.099469 1157 ts_tablet_manager.cc:1939] T fbcadec0c3184feaad4a883e5588f310 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.099599 1166 ts_tablet_manager.cc:1918] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.099763 1166 log.cc:1198] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/7d7539cf9ece4befaae6c521d471ea57
I20250109 20:52:54.099721 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9a4ad54fef41c3941fd9a02be5a418 (DEFAULT_TABLE table=default.Table_ad073c54c6aa42579dde0692025efcef [id=560a2c7adf7e45bfa286b1d6ad6e3083]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.100116 1166 ts_tablet_manager.cc:1939] T 7d7539cf9ece4befaae6c521d471ea57 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:54.100204 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9a4ad54fef41c3941fd9a02be5a418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.100425 1157 tablet_replica.cc:331] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.100602 1157 raft_consensus.cc:2238] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.100798 1157 raft_consensus.cc:2267] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.101469 1166 tablet_replica.cc:331] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:54.101939 1157 ts_tablet_manager.cc:1905] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.095662 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bc43af02a045473cac4e0b21c2dfc318" } new_table_name: "default.Table_19a1e320401641a383cf7c3fc20b4713" modify_external_catalogs: true
W20250109 20:52:54.104316 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7420a91ef4554611bfbb139ea53bd075 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.107244 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet fbcadec0c3184feaad4a883e5588f310 (table default.Table_e951676901de4adab1fb93a2d265da4f [id=97659ab8e1774179b7854d0985954b0b]) successfully deleted
I20250109 20:52:54.107502 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 7d7539cf9ece4befaae6c521d471ea57 (table default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]) successfully deleted
I20250109 20:52:54.107846 1166 raft_consensus.cc:2238] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.108105 1166 raft_consensus.cc:2267] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.109043 1176 tablet_bootstrap.cc:492] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.109066 1166 ts_tablet_manager.cc:1905] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.109169 1176 ts_tablet_manager.cc:1397] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.005s
I20250109 20:52:54.109331 1157 ts_tablet_manager.cc:1918] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.109478 1157 log.cc:1198] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/e64207928ef141118e7fc8fe7bb2379b
I20250109 20:52:54.109827 1157 ts_tablet_manager.cc:1939] T e64207928ef141118e7fc8fe7bb2379b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.109936 1176 raft_consensus.cc:357] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.110128 1176 raft_consensus.cc:383] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.110190 1176 raft_consensus.cc:738] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.110374 1176 consensus_queue.cc:260] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.111395 1176 ts_tablet_manager.cc:1428] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:54.111613 1176 tablet_bootstrap.cc:492] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:54.113059 1176 tablet_bootstrap.cc:654] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.115819 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet e64207928ef141118e7fc8fe7bb2379b (table default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]) successfully deleted
I20250109 20:52:54.116238 1157 tablet_replica.cc:331] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.116557 1157 raft_consensus.cc:2238] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.116811 1157 raft_consensus.cc:2267] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.118764 739 tablet_service.cc:1467] Processing CreateTablet for tablet bdce283727d1491ba0968a17c1c00d87 (DEFAULT_TABLE table=default.Table_028d1b883d75407dbfa05dc1c7ba26cf [id=b33580bb9b3b4ffda90b3f3d49f9f388]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.118956 1033 tablet_service.cc:1467] Processing CreateTablet for tablet e7293935c0f346e98ec8117cf1887868 (DEFAULT_TABLE table=default.Table_c21468e8f0b14e899d210feae2ef211e [id=b9b1a10b5da14d54aa1da0452920c1ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.119148 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdce283727d1491ba0968a17c1c00d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.119323 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7293935c0f346e98ec8117cf1887868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.121042 1029 tablet_service.cc:1467] Processing CreateTablet for tablet bdce283727d1491ba0968a17c1c00d87 (DEFAULT_TABLE table=default.Table_028d1b883d75407dbfa05dc1c7ba26cf [id=b33580bb9b3b4ffda90b3f3d49f9f388]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.120960 746 tablet_service.cc:1467] Processing CreateTablet for tablet e7293935c0f346e98ec8117cf1887868 (DEFAULT_TABLE table=default.Table_c21468e8f0b14e899d210feae2ef211e [id=b9b1a10b5da14d54aa1da0452920c1ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.121358 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdce283727d1491ba0968a17c1c00d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.121590 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7293935c0f346e98ec8117cf1887868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.121279 1157 ts_tablet_manager.cc:1905] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.138425 1176 tablet_bootstrap.cc:492] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.138546 1176 ts_tablet_manager.cc:1397] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.006s
I20250109 20:52:54.139256 1176 raft_consensus.cc:357] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.139405 1176 raft_consensus.cc:383] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.139441 1176 raft_consensus.cc:738] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.139585 1176 consensus_queue.cc:260] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.140283 1176 ts_tablet_manager.cc:1428] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:54.140571 1176 tablet_bootstrap.cc:492] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:54.142356 1176 tablet_bootstrap.cc:654] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:54.146050 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41583, user_credentials={real_user=slave}} blocked reactor thread for 173690us
I20250109 20:52:54.148762 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.150100 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 2be32f590fd844b2bbff91488a5390fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.150964 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 65570e42a9bd45bcba0a82abd12a9356 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.151775 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 540531463c184b9abf226d7487939e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.152021 1160 tablet_replica.cc:331] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:54.152227 1160 raft_consensus.cc:2238] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.152385 1160 raft_consensus.cc:2267] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.153240 1160 ts_tablet_manager.cc:1905] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.154198 1157 ts_tablet_manager.cc:1918] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.154373 1157 log.cc:1198] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/46d4cad7594346dfafdee8e4c49d74e6
I20250109 20:52:54.154743 1157 ts_tablet_manager.cc:1939] T 46d4cad7594346dfafdee8e4c49d74e6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.155788 1157 tablet_replica.cc:331] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.156014 1157 raft_consensus.cc:2238] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.156184 1157 raft_consensus.cc:2267] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.157089 1157 ts_tablet_manager.cc:1905] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.157235 1176 tablet_bootstrap.cc:492] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.157258 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 46d4cad7594346dfafdee8e4c49d74e6 (table default.Table_49b7725a1f7647cd9b136c55bd8ed301 [id=ccd17bbcb44f49a3b2eae73c6ace6c4c]) successfully deleted
I20250109 20:52:54.157320 1176 ts_tablet_manager.cc:1397] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20250109 20:52:54.158660 1176 raft_consensus.cc:357] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.158812 1176 raft_consensus.cc:383] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.158869 1176 raft_consensus.cc:738] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.159082 1176 consensus_queue.cc:260] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.160200 1176 ts_tablet_manager.cc:1428] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:54.160403 1176 tablet_bootstrap.cc:492] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:54.162889 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 with delete_type TABLET_DATA_DELETED (replaced by tablet d79437d800f44c5a8222eea7dc2cd4e3) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.162976 1157 ts_tablet_manager.cc:1918] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.163082 1157 log.cc:1198] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ecb55a61d2e94cbb88d2f44fff1aa8d9
I20250109 20:52:54.163448 1157 ts_tablet_manager.cc:1939] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.163456 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 712acab92e0f44a3a585d7d871498aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e020eeb03e041ed977779a07de5bb80) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.163935 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cdf5b737504d4e8f598cd2b265192e with delete_type TABLET_DATA_DELETED (replaced by tablet 1591374d521a47ebb405a812fe656b19) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.164029 1176 tablet_bootstrap.cc:654] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.164335 1157 tablet_replica.cc:331] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.164557 1157 raft_consensus.cc:2238] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.164611 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 (table default.Table_58060dd58dce4f7a8afa4d8c891c3ee3 [id=9a08f00bc4e74fc5861932c6f7cbbc58]) successfully deleted
I20250109 20:52:54.164748 1157 raft_consensus.cc:2267] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.165222 896 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9a4ad54fef41c3941fd9a02be5a418 (DEFAULT_TABLE table=default.Table_ad073c54c6aa42579dde0692025efcef [id=560a2c7adf7e45bfa286b1d6ad6e3083]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.166636 893 tablet_service.cc:1467] Processing CreateTablet for tablet e7293935c0f346e98ec8117cf1887868 (DEFAULT_TABLE table=default.Table_c21468e8f0b14e899d210feae2ef211e [id=b9b1a10b5da14d54aa1da0452920c1ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.167735 891 tablet_service.cc:1467] Processing CreateTablet for tablet bdce283727d1491ba0968a17c1c00d87 (DEFAULT_TABLE table=default.Table_028d1b883d75407dbfa05dc1c7ba26cf [id=b33580bb9b3b4ffda90b3f3d49f9f388]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.168983 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9a4ad54fef41c3941fd9a02be5a418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.171054 1176 tablet_bootstrap.cc:492] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.171160 1176 ts_tablet_manager.cc:1397] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20250109 20:52:54.171845 1176 raft_consensus.cc:357] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.172008 1176 raft_consensus.cc:383] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.172055 1176 raft_consensus.cc:738] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.172212 1176 consensus_queue.cc:260] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.172641 1176 ts_tablet_manager.cc:1428] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:52:54.172876 1176 tablet_bootstrap.cc:492] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:54.172958 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdce283727d1491ba0968a17c1c00d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.173600 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7293935c0f346e98ec8117cf1887868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.174288 1176 tablet_bootstrap.cc:654] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.174422 1160 ts_tablet_manager.cc:1918] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.174618 1160 log.cc:1198] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/540531463c184b9abf226d7487939e60
I20250109 20:52:54.175081 1160 ts_tablet_manager.cc:1939] T 540531463c184b9abf226d7487939e60 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:54.183795 1157 ts_tablet_manager.cc:1905] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.186527 1176 tablet_bootstrap.cc:492] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.186647 1176 ts_tablet_manager.cc:1397] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20250109 20:52:54.187309 1176 raft_consensus.cc:357] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.187475 1176 raft_consensus.cc:383] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.187532 1176 raft_consensus.cc:738] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.189245 1176 consensus_queue.cc:260] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.190908 1176 ts_tablet_manager.cc:1428] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:54.191125 1176 tablet_bootstrap.cc:492] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:54.192551 1176 tablet_bootstrap.cc:654] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.199579 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b9f5b6ca7ef84234b3d667d50ee5b8cd
I20250109 20:52:54.200902 1157 ts_tablet_manager.cc:1918] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.201066 1157 log.cc:1198] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/402280f75e51463a865c0fb521d0fd8a
I20250109 20:52:54.201431 1157 ts_tablet_manager.cc:1939] T 402280f75e51463a865c0fb521d0fd8a P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.201742 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.201808 746 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.202804 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 402280f75e51463a865c0fb521d0fd8a (table default.Table_e740022397a8451e830a159fa021e461 [id=24e621d6ed7e4a78a53808f4a8c2dd23]) successfully deleted
I20250109 20:52:54.203244 1157 tablet_replica.cc:331] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.203673 1157 raft_consensus.cc:2238] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.203919 1157 raft_consensus.cc:2267] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.204579 1157 ts_tablet_manager.cc:1905] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.205457 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.206091 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0f68bc0a564c4210bb29172a8759ebf3
I20250109 20:52:54.206698 454 catalog_manager.cc:6596] ReplaceTablet: tablet 0f68bc0a564c4210bb29172a8759ebf3 deleted and replaced by tablet 5a5f945b345d4c029798756e63867156
I20250109 20:52:54.206832 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f68bc0a564c4210bb29172a8759ebf3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5f945b345d4c029798756e63867156) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.207027 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b856f34058994714afb4ce41d44f5384" } modify_external_catalogs: true
I20250109 20:52:54.207208 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b856f34058994714afb4ce41d44f5384" } modify_external_catalogs: true
I20250109 20:52:54.208420 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f68bc0a564c4210bb29172a8759ebf3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5f945b345d4c029798756e63867156) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.212401 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_81566c8f4bd940aab3c5fca6ab8ff292" } modify_external_catalogs: true
I20250109 20:52:54.212596 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_81566c8f4bd940aab3c5fca6ab8ff292" } modify_external_catalogs: true
I20250109 20:52:54.214015 1157 ts_tablet_manager.cc:1918] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.214193 1157 log.cc:1198] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/cba68160be6145ddbbd2019c9b5f0076
I20250109 20:52:54.214685 1157 ts_tablet_manager.cc:1939] T cba68160be6145ddbbd2019c9b5f0076 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.215823 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet cba68160be6145ddbbd2019c9b5f0076 (table default.Table_b70135c5a9b548538e791703e28af8f7 [id=f556fe2500bb47efba4056c36ca34823]) successfully deleted
I20250109 20:52:54.216363 1157 tablet_replica.cc:331] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.217274 1157 raft_consensus.cc:2238] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:54.217603 1157 pending_rounds.cc:62] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Trying to abort 1 pending ops.
I20250109 20:52:54.218189 1806 raft_consensus.cc:491] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.218307 1806 raft_consensus.cc:513] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.218900 1806 leader_election.cc:290] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.219234 746 tablet_service.cc:1467] Processing CreateTablet for tablet cca642fb9c3e4a55adfb6ad8386a67cd (DEFAULT_TABLE table=default.Table_63fb1754d95144a3877b430bab3337b6 [id=cf501467078c4e948fa07d0f7121db66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.220209 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881ea20c17904d669c8a1fbd766ba559" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.220592 755 tablet_service.cc:1467] Processing CreateTablet for tablet 58f3afa4595a41baa77723f14d6a4944 (DEFAULT_TABLE table=default.Table_603546c0274d4fb49ab48d08448c658d [id=8a877abacb8c4c6fac2a4d9a175b8483]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:54.220664 989 leader_election.cc:343] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.221726 1033 tablet_service.cc:1467] Processing CreateTablet for tablet cca642fb9c3e4a55adfb6ad8386a67cd (DEFAULT_TABLE table=default.Table_63fb1754d95144a3877b430bab3337b6 [id=cf501467078c4e948fa07d0f7121db66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.221732 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 3e67d92bb9034dfaacdca9cbcea57550 (DEFAULT_TABLE table=default.Table_f5bea97ad34a41b1a53d274d1c918082 [id=a599c0f9354a42278a5c4f0d7c884412]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.222149 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca642fb9c3e4a55adfb6ad8386a67cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.223570 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca642fb9c3e4a55adfb6ad8386a67cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.223600 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881ea20c17904d669c8a1fbd766ba559" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:54.224081 990 leader_election.cc:343] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.224170 990 leader_election.cc:304] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.224527 1806 raft_consensus.cc:2743] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.224752 739 tablet_service.cc:1467] Processing CreateTablet for tablet 3e67d92bb9034dfaacdca9cbcea57550 (DEFAULT_TABLE table=default.Table_f5bea97ad34a41b1a53d274d1c918082 [id=a599c0f9354a42278a5c4f0d7c884412]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.224874 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58f3afa4595a41baa77723f14d6a4944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.225121 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e67d92bb9034dfaacdca9cbcea57550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.225139 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e67d92bb9034dfaacdca9cbcea57550. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:54.230130 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 -> Peer 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583): Couldn't send request to peer 36a29ab2002d4eb389cd11778988e1e9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250109 20:52:54.230860 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 246fd9ac96f54d008e8203e4580e7ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 246fd9ac96f54d008e8203e4580e7ef3 already in progress: creating tablet
W20250109 20:52:54.231149 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2be32f590fd844b2bbff91488a5390fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2be32f590fd844b2bbff91488a5390fc already in progress: creating tablet
W20250109 20:52:54.231359 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 65570e42a9bd45bcba0a82abd12a9356 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65570e42a9bd45bcba0a82abd12a9356 already in progress: creating tablet
I20250109 20:52:54.231835 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 540531463c184b9abf226d7487939e60 (table default.Table_642aeae240da4ea3a0fca064e8489134 [id=59a36af89f8f4df0ace607f3190236ba]) successfully deleted
I20250109 20:52:54.231877 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3e67d92bb9034dfaacdca9cbcea57550 (DEFAULT_TABLE table=default.Table_f5bea97ad34a41b1a53d274d1c918082 [id=a599c0f9354a42278a5c4f0d7c884412]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:54.232051 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 728e9b6101e3447fb3c24b3917aa4872 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 728e9b6101e3447fb3c24b3917aa4872 already in progress: creating tablet
I20250109 20:52:54.232165 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:54.232270 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 712acab92e0f44a3a585d7d871498aa3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 712acab92e0f44a3a585d7d871498aa3 already in progress: creating tablet
W20250109 20:52:54.232457 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 75cdf5b737504d4e8f598cd2b265192e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75cdf5b737504d4e8f598cd2b265192e already in progress: creating tablet
I20250109 20:52:54.232291 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e67d92bb9034dfaacdca9cbcea57550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.232656 885 tablet_service.cc:1467] Processing CreateTablet for tablet 58f3afa4595a41baa77723f14d6a4944 (DEFAULT_TABLE table=default.Table_603546c0274d4fb49ab48d08448c658d [id=8a877abacb8c4c6fac2a4d9a175b8483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.233023 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58f3afa4595a41baa77723f14d6a4944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.235628 1165 tablet_bootstrap.cc:492] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:54.235750 1165 ts_tablet_manager.cc:1397] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.291s user 0.039s sys 0.032s
I20250109 20:52:54.236346 1165 raft_consensus.cc:357] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.236528 1165 raft_consensus.cc:383] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.236584 1165 raft_consensus.cc:738] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.236802 1165 consensus_queue.cc:260] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.237305 1165 ts_tablet_manager.cc:1428] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:52:54.237946 1165 tablet_bootstrap.cc:492] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:54.239243 1165 tablet_bootstrap.cc:654] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.244186 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 89719a0d926c458d825371782980468c
I20250109 20:52:54.246441 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 89719a0d926c458d825371782980468c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.247363 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 89719a0d926c458d825371782980468c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.247756 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 58f3afa4595a41baa77723f14d6a4944 (DEFAULT_TABLE table=default.Table_603546c0274d4fb49ab48d08448c658d [id=8a877abacb8c4c6fac2a4d9a175b8483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.248204 751 tablet_service.cc:1467] Processing CreateTablet for tablet aba42482feab4d0f85fb46b4c8b01d19 (DEFAULT_TABLE table=default.Table_95809365f485432a9035b1a4a42d6dae [id=3875ae127e6e40268c5354584b3169e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.248664 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba42482feab4d0f85fb46b4c8b01d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.249850 1016 tablet_service.cc:1467] Processing CreateTablet for tablet a0cf3e606c6e403793fe7ff8e0796efa (DEFAULT_TABLE table=default.Table_6cba456285b14fd7921032996e6e4c89 [id=ad300680de22496aab366860b449339f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.251991 1166 ts_tablet_manager.cc:1918] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.252090 1166 log.cc:1198] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b221e048f6f6463e84e630386c304a05
I20250109 20:52:54.252383 1166 ts_tablet_manager.cc:1939] T b221e048f6f6463e84e630386c304a05 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:54.253002 753 tablet_service.cc:1467] Processing CreateTablet for tablet f0d09d4894634825ad3aa7933bfb6010 (DEFAULT_TABLE table=default.Table_ff034ea8af80440fbe26a4c05c434f2c [id=359c96626b9f4a2a851a03f88f98498b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.253351 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d09d4894634825ad3aa7933bfb6010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.257000 1035 tablet_service.cc:1467] Processing CreateTablet for tablet f0d09d4894634825ad3aa7933bfb6010 (DEFAULT_TABLE table=default.Table_ff034ea8af80440fbe26a4c05c434f2c [id=359c96626b9f4a2a851a03f88f98498b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.257458 754 tablet_service.cc:1467] Processing CreateTablet for tablet 158486f4929e419cbba9d6132c8136ab (DEFAULT_TABLE table=default.Table_3219be93dffd45bb89a8ac2f801d567a [id=25198238432849c186be79c6f7d0dbb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.257807 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 158486f4929e419cbba9d6132c8136ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.260353 1806 raft_consensus.cc:491] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.260455 1806 raft_consensus.cc:513] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.261018 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 158486f4929e419cbba9d6132c8136ab (DEFAULT_TABLE table=default.Table_3219be93dffd45bb89a8ac2f801d567a [id=25198238432849c186be79c6f7d0dbb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.261188 1806 leader_election.cc:290] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.261552 750 tablet_service.cc:1467] Processing CreateTablet for tablet a0cf3e606c6e403793fe7ff8e0796efa (DEFAULT_TABLE table=default.Table_6cba456285b14fd7921032996e6e4c89 [id=ad300680de22496aab366860b449339f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.261899 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cf3e606c6e403793fe7ff8e0796efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.262137 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 2a83e584f7694f5e9407449f73f9c907 (DEFAULT_TABLE table=default.Table_5e1c8b41b7ee4ac08260bfdefdc0555f [id=762d5ee242914bcda632f360fefd011b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.262521 1176 tablet_bootstrap.cc:492] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.262647 1176 ts_tablet_manager.cc:1397] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.017s
I20250109 20:52:54.263513 1176 raft_consensus.cc:357] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.263712 1176 raft_consensus.cc:383] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.263774 1176 raft_consensus.cc:738] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.265100 744 tablet_service.cc:1467] Processing CreateTablet for tablet 2a83e584f7694f5e9407449f73f9c907 (DEFAULT_TABLE table=default.Table_5e1c8b41b7ee4ac08260bfdefdc0555f [id=762d5ee242914bcda632f360fefd011b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.265233 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 741d13db17e847039d3f9e25cafc6db0 (DEFAULT_TABLE table=default.Table_a82f87794b0d48f09f851336ce4e89d5 [id=65b1953277304052921145c8850bea29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.265506 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a83e584f7694f5e9407449f73f9c907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.265861 1176 consensus_queue.cc:260] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.266414 1157 pending_rounds.cc:69] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523669625364480 op_type: NO_OP noop_request { }
I20250109 20:52:54.267884 1176 ts_tablet_manager.cc:1428] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:54.266554 1157 raft_consensus.cc:2883] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:54.268364 1157 raft_consensus.cc:2267] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.268976 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff04093442e413abe8433e8784b254f (DEFAULT_TABLE table=default.Table_e5d7cc7fe84d481a974efb955fbeb6e4 [id=1ee9b4458d1140c99cc46f7ec3fa1213]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.261914 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b47656fe1ae4c9ba4138bcd37c0e6db" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:54.269677 990 leader_election.cc:343] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.267572 757 tablet_service.cc:1467] Processing CreateTablet for tablet 56fac585b2904ab490a6882dd5252a9a (DEFAULT_TABLE table=default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.270171 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fac585b2904ab490a6882dd5252a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.271097 1157 ts_tablet_manager.cc:1905] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.266755 1158 tablet_bootstrap.cc:492] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:54.271340 1158 ts_tablet_manager.cc:1397] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.303s user 0.054s sys 0.112s
W20250109 20:52:54.272377 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 9558dfe563d94482bf496e19b8a80edc (table default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9558dfe563d94482bf496e19b8a80edc consensus state has no leader
I20250109 20:52:54.272951 1032 tablet_service.cc:1467] Processing CreateTablet for tablet aba42482feab4d0f85fb46b4c8b01d19 (DEFAULT_TABLE table=default.Table_95809365f485432a9035b1a4a42d6dae [id=3875ae127e6e40268c5354584b3169e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.273464 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b47656fe1ae4c9ba4138bcd37c0e6db" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:54.274540 989 leader_election.cc:343] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.274639 989 leader_election.cc:304] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.275003 1806 raft_consensus.cc:2743] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.273686 885 tablet_service.cc:1467] Processing CreateTablet for tablet aba42482feab4d0f85fb46b4c8b01d19 (DEFAULT_TABLE table=default.Table_95809365f485432a9035b1a4a42d6dae [id=3875ae127e6e40268c5354584b3169e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.273954 897 tablet_service.cc:1467] Processing CreateTablet for tablet a0cf3e606c6e403793fe7ff8e0796efa (DEFAULT_TABLE table=default.Table_6cba456285b14fd7921032996e6e4c89 [id=ad300680de22496aab366860b449339f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.275492 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cf3e606c6e403793fe7ff8e0796efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.275622 1810 raft_consensus.cc:491] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.275732 1810 raft_consensus.cc:513] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.276285 1810 leader_election.cc:290] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.276677 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c367a8ca184dfa9e4b11094ba76140" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:54.277112 712 leader_election.cc:343] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.274222 886 tablet_service.cc:1467] Processing CreateTablet for tablet 2a83e584f7694f5e9407449f73f9c907 (DEFAULT_TABLE table=default.Table_5e1c8b41b7ee4ac08260bfdefdc0555f [id=762d5ee242914bcda632f360fefd011b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.281531 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a83e584f7694f5e9407449f73f9c907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.274515 894 tablet_service.cc:1467] Processing CreateTablet for tablet 741d13db17e847039d3f9e25cafc6db0 (DEFAULT_TABLE table=default.Table_a82f87794b0d48f09f851336ce4e89d5 [id=65b1953277304052921145c8850bea29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.287747 1157 ts_tablet_manager.cc:1918] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:54.287784 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741d13db17e847039d3f9e25cafc6db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.287873 1157 log.cc:1198] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4d8e7081cac440389bec5dfac2391beb
I20250109 20:52:54.288333 1157 ts_tablet_manager.cc:1939] T 4d8e7081cac440389bec5dfac2391beb P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.289366 1157 tablet_replica.cc:331] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.289587 1157 raft_consensus.cc:2238] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.289779 1157 raft_consensus.cc:2267] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.290629 887 tablet_service.cc:1467] Processing CreateTablet for tablet 158486f4929e419cbba9d6132c8136ab (DEFAULT_TABLE table=default.Table_3219be93dffd45bb89a8ac2f801d567a [id=25198238432849c186be79c6f7d0dbb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.291111 893 tablet_service.cc:1467] Processing CreateTablet for tablet f0d09d4894634825ad3aa7933bfb6010 (DEFAULT_TABLE table=default.Table_ff034ea8af80440fbe26a4c05c434f2c [id=359c96626b9f4a2a851a03f88f98498b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.291383 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d09d4894634825ad3aa7933bfb6010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.274809 882 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff04093442e413abe8433e8784b254f (DEFAULT_TABLE table=default.Table_e5d7cc7fe84d481a974efb955fbeb6e4 [id=1ee9b4458d1140c99cc46f7ec3fa1213]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.275108 892 tablet_service.cc:1467] Processing CreateTablet for tablet 56fac585b2904ab490a6882dd5252a9a (DEFAULT_TABLE table=default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.293304 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fac585b2904ab490a6882dd5252a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.296494 891 tablet_service.cc:1467] Processing CreateTablet for tablet cca642fb9c3e4a55adfb6ad8386a67cd (DEFAULT_TABLE table=default.Table_63fb1754d95144a3877b430bab3337b6 [id=cf501467078c4e948fa07d0f7121db66]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:54.297327 713 leader_election.cc:343] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: e4c367a8ca184dfa9e4b11094ba76140
I20250109 20:52:54.297487 713 leader_election.cc:304] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:54.298162 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58f3afa4595a41baa77723f14d6a4944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.298506 1810 raft_consensus.cc:2743] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.300901 1806 raft_consensus.cc:491] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.301030 1176 tablet_bootstrap.cc:492] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:54.301021 1806 raft_consensus.cc:513] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.301545 1806 leader_election.cc:290] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:54.301589 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cf3e606c6e403793fe7ff8e0796efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.302063 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741d13db17e847039d3f9e25cafc6db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.302621 1176 tablet_bootstrap.cc:654] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.302834 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff04093442e413abe8433e8784b254f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.303290 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a83e584f7694f5e9407449f73f9c907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.303777 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7dc6809d6547f6b0354b64d7149bf4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:54.305788 990 leader_election.cc:343] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.306006 1806 raft_consensus.cc:491] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.306069 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d09d4894634825ad3aa7933bfb6010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.306143 1806 raft_consensus.cc:513] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.306510 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 158486f4929e419cbba9d6132c8136ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.306761 1806 leader_election.cc:290] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:54.306968 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba42482feab4d0f85fb46b4c8b01d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.313494 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4d8e7081cac440389bec5dfac2391beb (table default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3 [id=e1afc4c2550f49f4b3c7b17ea2f09f60]) successfully deleted
I20250109 20:52:54.313670 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a455033b674e5a94fc37d0275eb313" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.315085 741 tablet_service.cc:1467] Processing CreateTablet for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd (DEFAULT_TABLE table=default.Table_1cf2c16f6ff048719047b27308f7cf99 [id=c51c57c366f34948a2cf25fff061e4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.315526 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.315704 757 tablet_service.cc:1467] Processing CreateTablet for tablet 89719a0d926c458d825371782980468c (DEFAULT_TABLE table=default.Table_036a876d87ae4027b9002e151ed56074 [id=307bd3e5b1ec45598c9e5804195e8fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.316349 750 tablet_service.cc:1467] Processing CreateTablet for tablet 29f1978ebf88470b80b8d4e38c6bc609 (DEFAULT_TABLE table=default.Table_cd252806276c4bc28ada21e4c72c6c08 [id=d341484e994c40b3990935db8ac1c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.316924 753 tablet_service.cc:1467] Processing CreateTablet for tablet 3774aacd532242b788a7af6044dde6b3 (DEFAULT_TABLE table=default.Table_3b113c9e02ff40f2b63eab79960b09bd [id=3d2ef43007ac4f0aaa8e36bc39cd4b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.317449 754 tablet_service.cc:1467] Processing CreateTablet for tablet b920fbb432e64b2ea8891e4ca8966de9 (DEFAULT_TABLE table=default.Table_9151edfc060b4199b6ca8ec93a8ab836 [id=f34eb90407bb40028d72035bd901fdfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.267988 755 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff04093442e413abe8433e8784b254f (DEFAULT_TABLE table=default.Table_e5d7cc7fe84d481a974efb955fbeb6e4 [id=1ee9b4458d1140c99cc46f7ec3fa1213]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.318642 744 tablet_service.cc:1467] Processing CreateTablet for tablet 86276d5c75be47e1bc5763ee532ad8ab (DEFAULT_TABLE table=default.Table_c918439b959a4a4fa4f2c75750ff33ca [id=2b4816aba5a94cb0805ea85c94cc2de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.319152 751 tablet_service.cc:1467] Processing CreateTablet for tablet 6787f74100894fe6a7053a6d92ea7b96 (DEFAULT_TABLE table=default.Table_b1b97f9ea66f4c7c89289094e6d57533 [id=d792d447bf93470bb29ae0b9423783ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.316367 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89719a0d926c458d825371782980468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.296847 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff04093442e413abe8433e8784b254f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.322999 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f1978ebf88470b80b8d4e38c6bc609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.303272 1771 raft_consensus.cc:491] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.324836 1771 raft_consensus.cc:513] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.325480 1771 leader_election.cc:290] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.303432 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7dc6809d6547f6b0354b64d7149bf4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.326454 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f494150713b4dde9399d53d57567983" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:54.326649 1055 raft_consensus.cc:2388] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d473955919435cb679edd5f8afe2c2 in current term 1: Already voted for candidate 09833ea46b7141d58c3899dace69e49f in this term.
I20250109 20:52:54.327494 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f494150713b4dde9399d53d57567983" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.327973 854 leader_election.cc:304] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:54.328408 1771 raft_consensus.cc:3054] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.303962 1165 tablet_bootstrap.cc:492] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:54.308867 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a455033b674e5a94fc37d0275eb313" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.328878 1165 ts_tablet_manager.cc:1397] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.091s user 0.012s sys 0.001s
I20250109 20:52:54.328990 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 158486f4929e419cbba9d6132c8136ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.329500 1165 raft_consensus.cc:357] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.329644 1165 raft_consensus.cc:383] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.329684 1165 raft_consensus.cc:738] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.329856 1165 consensus_queue.cc:260] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.330271 1165 ts_tablet_manager.cc:1428] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:52:54.330479 1165 tablet_bootstrap.cc:492] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:54.330703 1771 raft_consensus.cc:2743] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.330749 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca642fb9c3e4a55adfb6ad8386a67cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:54.331282 989 leader_election.cc:343] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.331385 989 leader_election.cc:304] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.331785 1165 tablet_bootstrap.cc:654] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.331938 1806 raft_consensus.cc:2743] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.332959 748 tablet_service.cc:1467] Processing CreateTablet for tablet 73f948716bfa408193144275d7c9f5ab (DEFAULT_TABLE table=default.Table_d2a69f7c330144e6bd09c7df8c65821e [id=8773098b8c5c4989aa76c7e1219805ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.333360 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f948716bfa408193144275d7c9f5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.335254 752 tablet_service.cc:1467] Processing CreateTablet for tablet 36bc408ae5174be89dbc2a13de464dd4 (DEFAULT_TABLE table=default.Table_76d70836419947c78b06922a4fe21a2c [id=53718e55e4344456851f408909cc8170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.335642 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bc408ae5174be89dbc2a13de464dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.337280 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1132c08b23ea4849a78b4809cbe44360 (DEFAULT_TABLE table=default.Table_96de890f59cf48cb8d0b23bb9d8063fe [id=9bfecb3926ef48c9a99e28a92e1bc284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.337632 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1132c08b23ea4849a78b4809cbe44360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.342675 743 tablet_service.cc:1467] Processing CreateTablet for tablet c24377e7b6354110a36711c69a878c58 (DEFAULT_TABLE table=default.Table_5c4d17cdde7b4233916baefff36aeb42 [id=b4bc0b87a0f04907a416d31068ae2def]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.342974 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba42482feab4d0f85fb46b4c8b01d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.343010 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24377e7b6354110a36711c69a878c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.345388 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3974c4196d4043be9ad6e0452ed9cb7b (DEFAULT_TABLE table=default.Table_3a269f56a3b446c1973d3b9a24f0ec8f [id=16cb6983d685473c8d7aba4e39573aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.345705 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3974c4196d4043be9ad6e0452ed9cb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.347796 752 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd758fa2fa4cebbdf7faf3856f1941 (DEFAULT_TABLE table=default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.348135 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bd758fa2fa4cebbdf7faf3856f1941. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.348932 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff04093442e413abe8433e8784b254f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.349304 745 tablet_service.cc:1467] Processing CreateTablet for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 (DEFAULT_TABLE table=default.Table_6c6662f9bad8425db77969d4150d9c38 [id=50f790f5c8f243ba9c12c2a6bd515608]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.349655 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ebf92383a5d419ba3e24ea4eea7ee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.352888 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3774aacd532242b788a7af6044dde6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.353262 756 tablet_service.cc:1467] Processing CreateTablet for tablet 86a59ce429b847f1b035b4dd06e39f70 (DEFAULT_TABLE table=default.Table_2ea95ef93df94b41a15c6f2ce7252567 [id=d55ef52b57c346acb6e5fea55de715fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.353582 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86a59ce429b847f1b035b4dd06e39f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.265502 746 tablet_service.cc:1467] Processing CreateTablet for tablet 741d13db17e847039d3f9e25cafc6db0 (DEFAULT_TABLE table=default.Table_a82f87794b0d48f09f851336ce4e89d5 [id=65b1953277304052921145c8850bea29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.354209 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741d13db17e847039d3f9e25cafc6db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.354779 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b920fbb432e64b2ea8891e4ca8966de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.357359 758 tablet_service.cc:1467] Processing CreateTablet for tablet 3997167fc6ee4cd0ae2401aa8924f684 (DEFAULT_TABLE table=default.Table_a34657904fd9418f8760104fbcc555e1 [id=99a31bfd08624f70a4be6acef63631fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.357673 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3997167fc6ee4cd0ae2401aa8924f684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.358352 740 tablet_service.cc:1467] Processing CreateTablet for tablet a6006590b7764efc901bf51ca4e19ab3 (DEFAULT_TABLE table=default.Table_cf33c8d126254dd580ef3fb97ecad07f [id=ab636e0f2f5d495c9a60d3a03dbd2d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.358670 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6006590b7764efc901bf51ca4e19ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.363194 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86276d5c75be47e1bc5763ee532ad8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.364398 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6787f74100894fe6a7053a6d92ea7b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.369432 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f68bc0a564c4210bb29172a8759ebf3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5f945b345d4c029798756e63867156) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.369902 891 tablet_service.cc:1467] Processing CreateTablet for tablet 6787f74100894fe6a7053a6d92ea7b96 (DEFAULT_TABLE table=default.Table_b1b97f9ea66f4c7c89289094e6d57533 [id=d792d447bf93470bb29ae0b9423783ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.369902 896 tablet_service.cc:1467] Processing CreateTablet for tablet 86276d5c75be47e1bc5763ee532ad8ab (DEFAULT_TABLE table=default.Table_c918439b959a4a4fa4f2c75750ff33ca [id=2b4816aba5a94cb0805ea85c94cc2de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.371101 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86276d5c75be47e1bc5763ee532ad8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.374204 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6787f74100894fe6a7053a6d92ea7b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.374171 892 tablet_service.cc:1467] Processing CreateTablet for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd (DEFAULT_TABLE table=default.Table_1cf2c16f6ff048719047b27308f7cf99 [id=c51c57c366f34948a2cf25fff061e4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.374497 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.375612 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3774aacd532242b788a7af6044dde6b3 (DEFAULT_TABLE table=default.Table_3b113c9e02ff40f2b63eab79960b09bd [id=3d2ef43007ac4f0aaa8e36bc39cd4b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.375845 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3774aacd532242b788a7af6044dde6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.376928 894 tablet_service.cc:1467] Processing CreateTablet for tablet 89719a0d926c458d825371782980468c (DEFAULT_TABLE table=default.Table_036a876d87ae4027b9002e151ed56074 [id=307bd3e5b1ec45598c9e5804195e8fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.377168 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89719a0d926c458d825371782980468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.381356 748 tablet_service.cc:1467] Processing CreateTablet for tablet 49ba8f186a444c8a9cf01d3eb2036c49 (DEFAULT_TABLE table=default.Table_a34821f5c1e44d4cb26f0daa699282d0 [id=c2efe4d24cb84908b5e238c6aed23c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.370797 886 tablet_service.cc:1467] Processing CreateTablet for tablet 29f1978ebf88470b80b8d4e38c6bc609 (DEFAULT_TABLE table=default.Table_cd252806276c4bc28ada21e4c72c6c08 [id=d341484e994c40b3990935db8ac1c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.381659 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ba8f186a444c8a9cf01d3eb2036c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.372042 893 tablet_service.cc:1467] Processing CreateTablet for tablet 73f948716bfa408193144275d7c9f5ab (DEFAULT_TABLE table=default.Table_d2a69f7c330144e6bd09c7df8c65821e [id=8773098b8c5c4989aa76c7e1219805ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.381982 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f1978ebf88470b80b8d4e38c6bc609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.384199 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 89719a0d926c458d825371782980468c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.384590 1157 ts_tablet_manager.cc:1905] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.386029 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f948716bfa408193144275d7c9f5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.387650 1166 tablet_replica.cc:331] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:54.387867 1166 raft_consensus.cc:2238] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.388031 1166 raft_consensus.cc:2267] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.388625 1166 ts_tablet_manager.cc:1905] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.389120 752 tablet_service.cc:1467] Processing CreateTablet for tablet 08209ea49c894209abca6c0881f534f5 (DEFAULT_TABLE table=default.Table_ab28e2a04600484e858415ada1017c00 [id=746fc0d6436d4f3f8bceb504b5f27f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.390074 897 tablet_service.cc:1467] Processing CreateTablet for tablet 1132c08b23ea4849a78b4809cbe44360 (DEFAULT_TABLE table=default.Table_96de890f59cf48cb8d0b23bb9d8063fe [id=9bfecb3926ef48c9a99e28a92e1bc284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.390367 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1132c08b23ea4849a78b4809cbe44360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.390705 887 tablet_service.cc:1467] Processing CreateTablet for tablet 36bc408ae5174be89dbc2a13de464dd4 (DEFAULT_TABLE table=default.Table_76d70836419947c78b06922a4fe21a2c [id=53718e55e4344456851f408909cc8170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.390982 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bc408ae5174be89dbc2a13de464dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.392536 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08209ea49c894209abca6c0881f534f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.393564 899 tablet_service.cc:1467] Processing CreateTablet for tablet 49ba8f186a444c8a9cf01d3eb2036c49 (DEFAULT_TABLE table=default.Table_a34821f5c1e44d4cb26f0daa699282d0 [id=c2efe4d24cb84908b5e238c6aed23c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.393915 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ba8f186a444c8a9cf01d3eb2036c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.394013 1158 raft_consensus.cc:357] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.394222 1158 raft_consensus.cc:383] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250109 20:52:54.314306 990 leader_election.cc:343] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:54.398005 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 83723.4us
I20250109 20:52:54.398448 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 56fac585b2904ab490a6882dd5252a9a (DEFAULT_TABLE table=default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.398634 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 86276d5c75be47e1bc5763ee532ad8ab (DEFAULT_TABLE table=default.Table_c918439b959a4a4fa4f2c75750ff33ca [id=2b4816aba5a94cb0805ea85c94cc2de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.398931 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86276d5c75be47e1bc5763ee532ad8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.400883 449 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3]). Expected version 2 got 0
W20250109 20:52:54.401125 989 leader_election.cc:343] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.401800 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 6787f74100894fe6a7053a6d92ea7b96 (DEFAULT_TABLE table=default.Table_b1b97f9ea66f4c7c89289094e6d57533 [id=d792d447bf93470bb29ae0b9423783ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.404489 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fac585b2904ab490a6882dd5252a9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:54.407379 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ebf92383a5d419ba3e24ea4eea7ee72 consensus state has no leader
I20250109 20:52:54.411207 890 tablet_service.cc:1467] Processing CreateTablet for tablet c24377e7b6354110a36711c69a878c58 (DEFAULT_TABLE table=default.Table_5c4d17cdde7b4233916baefff36aeb42 [id=b4bc0b87a0f04907a416d31068ae2def]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.411545 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24377e7b6354110a36711c69a878c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.412007 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 3774aacd532242b788a7af6044dde6b3 (DEFAULT_TABLE table=default.Table_3b113c9e02ff40f2b63eab79960b09bd [id=3d2ef43007ac4f0aaa8e36bc39cd4b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.415033 889 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd758fa2fa4cebbdf7faf3856f1941 (DEFAULT_TABLE table=default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.415354 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bd758fa2fa4cebbdf7faf3856f1941. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.416409 1165 tablet_bootstrap.cc:492] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:54.416637 1165 ts_tablet_manager.cc:1397] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.086s user 0.002s sys 0.005s
I20250109 20:52:54.417279 1165 raft_consensus.cc:357] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.417449 1165 raft_consensus.cc:383] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.417511 1165 raft_consensus.cc:738] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.417677 1165 consensus_queue.cc:260] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.419009 1678 raft_consensus.cc:491] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.419142 1678 raft_consensus.cc:513] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.419737 1678 leader_election.cc:290] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.420667 1165 ts_tablet_manager.cc:1428] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:52:54.421437 1165 tablet_bootstrap.cc:492] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:54.422915 1165 tablet_bootstrap.cc:654] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.418210 880 tablet_service.cc:1467] Processing CreateTablet for tablet 3974c4196d4043be9ad6e0452ed9cb7b (DEFAULT_TABLE table=default.Table_3a269f56a3b446c1973d3b9a24f0ec8f [id=16cb6983d685473c8d7aba4e39573aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.423513 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3974c4196d4043be9ad6e0452ed9cb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.425297 885 tablet_service.cc:1467] Processing CreateTablet for tablet b920fbb432e64b2ea8891e4ca8966de9 (DEFAULT_TABLE table=default.Table_9151edfc060b4199b6ca8ec93a8ab836 [id=f34eb90407bb40028d72035bd901fdfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.425623 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b920fbb432e64b2ea8891e4ca8966de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.426462 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8654c154dab465d8991910698027c3f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:54.427044 712 leader_election.cc:343] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.434347 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7cbb3a94bca245ed8531800592512f15" } modify_external_catalogs: true
I20250109 20:52:54.434618 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7cbb3a94bca245ed8531800592512f15" } modify_external_catalogs: true
I20250109 20:52:54.435618 1021 tablet_service.cc:1467] Processing CreateTablet for tablet b920fbb432e64b2ea8891e4ca8966de9 (DEFAULT_TABLE table=default.Table_9151edfc060b4199b6ca8ec93a8ab836 [id=f34eb90407bb40028d72035bd901fdfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.436041 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 89719a0d926c458d825371782980468c (DEFAULT_TABLE table=default.Table_036a876d87ae4027b9002e151ed56074 [id=307bd3e5b1ec45598c9e5804195e8fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.436061 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b920fbb432e64b2ea8891e4ca8966de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.437892 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_89576f5c757a4cc788608386bc1f949b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.438493 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89576f5c757a4cc788608386bc1f949b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.442184 1771 raft_consensus.cc:491] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.442308 1771 raft_consensus.cc:513] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.442755 893 tablet_service.cc:1467] Processing CreateTablet for tablet 08209ea49c894209abca6c0881f534f5 (DEFAULT_TABLE table=default.Table_ab28e2a04600484e858415ada1017c00 [id=746fc0d6436d4f3f8bceb504b5f27f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.445573 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 73f948716bfa408193144275d7c9f5ab (DEFAULT_TABLE table=default.Table_d2a69f7c330144e6bd09c7df8c65821e [id=8773098b8c5c4989aa76c7e1219805ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.436672 1016 tablet_service.cc:1467] Processing CreateTablet for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd (DEFAULT_TABLE table=default.Table_1cf2c16f6ff048719047b27308f7cf99 [id=c51c57c366f34948a2cf25fff061e4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.446508 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 29f1978ebf88470b80b8d4e38c6bc609 (DEFAULT_TABLE table=default.Table_cd252806276c4bc28ada21e4c72c6c08 [id=d341484e994c40b3990935db8ac1c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.450769 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08209ea49c894209abca6c0881f534f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.442804 897 tablet_service.cc:1467] Processing CreateTablet for tablet 3997167fc6ee4cd0ae2401aa8924f684 (DEFAULT_TABLE table=default.Table_a34657904fd9418f8760104fbcc555e1 [id=99a31bfd08624f70a4be6acef63631fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.453032 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3997167fc6ee4cd0ae2401aa8924f684. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:54.455938 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06e615ce8e124cfe9432778b6b58e117 consensus state has no leader
I20250109 20:52:54.442857 1771 leader_election.cc:290] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.443077 890 tablet_service.cc:1467] Processing CreateTablet for tablet 86a59ce429b847f1b035b4dd06e39f70 (DEFAULT_TABLE table=default.Table_2ea95ef93df94b41a15c6f2ce7252567 [id=d55ef52b57c346acb6e5fea55de715fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.459347 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86a59ce429b847f1b035b4dd06e39f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.460363 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
I20250109 20:52:54.461939 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8e9fb830afe541f08539f0852b9b1c30" } new_table_name: "default.Table_e8e4034056dd4856a0bdb4c83d34b6d9" modify_external_catalogs: true
I20250109 20:52:54.464147 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4d2ad301c11943b69b59be18d6775513
I20250109 20:52:54.464676 453 catalog_manager.cc:6596] ReplaceTablet: tablet 4d2ad301c11943b69b59be18d6775513 deleted and replaced by tablet 73b1822cd5744b5197d8141fc66539a0
W20250109 20:52:54.443614 854 leader_election.cc:343] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 2a89a0b47d10414d9b6a9123af1f4ab3
I20250109 20:52:54.445128 887 tablet_service.cc:1467] Processing CreateTablet for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 (DEFAULT_TABLE table=default.Table_6c6662f9bad8425db77969d4150d9c38 [id=50f790f5c8f243ba9c12c2a6bd515608]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.445176 889 tablet_service.cc:1467] Processing CreateTablet for tablet a6006590b7764efc901bf51ca4e19ab3 (DEFAULT_TABLE table=default.Table_cf33c8d126254dd580ef3fb97ecad07f [id=ab636e0f2f5d495c9a60d3a03dbd2d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.466069 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ebf92383a5d419ba3e24ea4eea7ee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.467015 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a51ccde2af1e4323b06854cff3e6216c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.467657 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a51ccde2af1e4323b06854cff3e6216c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:54.468912 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 89719a0d926c458d825371782980468c because the tablet was not found. No further retry: Not found: Tablet not found: 89719a0d926c458d825371782980468c
I20250109 20:52:54.470103 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89719a0d926c458d825371782980468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.470618 989 leader_election.cc:304] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.471021 1806 raft_consensus.cc:2743] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.471050 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2d8fde33d419436ca5839413c9b448da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.471589 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d8fde33d419436ca5839413c9b448da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.472251 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6006590b7764efc901bf51ca4e19ab3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:54.472420 854 leader_election.cc:343] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2a89a0b47d10414d9b6a9123af1f4ab3
I20250109 20:52:54.472517 854 leader_election.cc:304] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:54.472918 1771 raft_consensus.cc:2743] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.475827 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b221e048f6f6463e84e630386c304a05 (table default.Table_43ed51a629174e3bbaad39c5d8f43ef7 [id=ce77658acdf34b7e90401b0982c045fe]) successfully deleted
I20250109 20:52:54.477128 1166 ts_tablet_manager.cc:1918] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.479091 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ffd3a1b1d3e44feda6dbaa8556e377fc
W20250109 20:52:54.479672 712 consensus_peers.cc:487] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:54.479826 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c24377e7b6354110a36711c69a878c58 (DEFAULT_TABLE table=default.Table_5c4d17cdde7b4233916baefff36aeb42 [id=b4bc0b87a0f04907a416d31068ae2def]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.480062 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 36bc408ae5174be89dbc2a13de464dd4 (DEFAULT_TABLE table=default.Table_76d70836419947c78b06922a4fe21a2c [id=53718e55e4344456851f408909cc8170]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.480697 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.481190 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2ad301c11943b69b59be18d6775513 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b1822cd5744b5197d8141fc66539a0) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.481855 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0a56e4ac9a02430886c64ffa9268293f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.482425 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a56e4ac9a02430886c64ffa9268293f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.483067 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 49ba8f186a444c8a9cf01d3eb2036c49 (DEFAULT_TABLE table=default.Table_a34821f5c1e44d4cb26f0daa699282d0 [id=c2efe4d24cb84908b5e238c6aed23c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.483716 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f7662dfa61fe4100a214839f938d4a58" } new_table_name: "default.Table_7318e12d39b649ceb1971008c72c893b" modify_external_catalogs: true
I20250109 20:52:54.485253 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3e2f412cd17540698bf3788a7fc9cca8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.485875 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e2f412cd17540698bf3788a7fc9cca8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.487381 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d80f0ae2bf3e419faf22aa448e220dfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.487888 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d80f0ae2bf3e419faf22aa448e220dfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.490792 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 1132c08b23ea4849a78b4809cbe44360 (DEFAULT_TABLE table=default.Table_96de890f59cf48cb8d0b23bb9d8063fe [id=9bfecb3926ef48c9a99e28a92e1bc284]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.494120 1166 log.cc:1198] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2d96a569e6be4e01a26918d115cfbcf9
W20250109 20:52:54.494808 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 139222us
I20250109 20:52:54.499459 1157 ts_tablet_manager.cc:1918] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.499677 1157 log.cc:1198] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f5a5815d9ef54e3e959f151ed08ae812
I20250109 20:52:54.500075 1157 ts_tablet_manager.cc:1939] T f5a5815d9ef54e3e959f151ed08ae812 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.500119 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3974c4196d4043be9ad6e0452ed9cb7b (DEFAULT_TABLE table=default.Table_3a269f56a3b446c1973d3b9a24f0ec8f [id=16cb6983d685473c8d7aba4e39573aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.501438 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f5a5815d9ef54e3e959f151ed08ae812 (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:52:54.503088 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd758fa2fa4cebbdf7faf3856f1941 (DEFAULT_TABLE table=default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.507699 1678 raft_consensus.cc:491] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.507845 1678 raft_consensus.cc:513] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.508467 1678 leader_election.cc:290] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.509385 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e615ce8e124cfe9432778b6b58e117" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:54.510018 712 leader_election.cc:343] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.529460 1806 raft_consensus.cc:491] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.534611 1157 tablet_replica.cc:331] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.534991 1157 raft_consensus.cc:2238] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.535254 1157 raft_consensus.cc:2267] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.536092 1157 ts_tablet_manager.cc:1905] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.538446 917 raft_consensus.cc:3054] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.542574 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3774aacd532242b788a7af6044dde6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.542836 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ba8f186a444c8a9cf01d3eb2036c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.544994 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f1978ebf88470b80b8d4e38c6bc609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.542402 1678 consensus_queue.cc:1035] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:54.559526 1165 tablet_bootstrap.cc:492] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:54.559661 1165 ts_tablet_manager.cc:1397] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.138s user 0.049s sys 0.071s
I20250109 20:52:54.560456 1165 raft_consensus.cc:357] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.560655 1165 raft_consensus.cc:383] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.560693 1165 raft_consensus.cc:738] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.560953 1165 consensus_queue.cc:260] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.561692 1165 ts_tablet_manager.cc:1428] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:54.561807 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 08209ea49c894209abca6c0881f534f5 (DEFAULT_TABLE table=default.Table_ab28e2a04600484e858415ada1017c00 [id=746fc0d6436d4f3f8bceb504b5f27f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.561940 1165 tablet_bootstrap.cc:492] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:54.562841 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3974c4196d4043be9ad6e0452ed9cb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.563078 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08209ea49c894209abca6c0881f534f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.563527 1165 tablet_bootstrap.cc:654] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.564548 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3997167fc6ee4cd0ae2401aa8924f684 (DEFAULT_TABLE table=default.Table_a34657904fd9418f8760104fbcc555e1 [id=99a31bfd08624f70a4be6acef63631fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.566881 1165 tablet_bootstrap.cc:492] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:54.567034 1165 ts_tablet_manager.cc:1397] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250109 20:52:54.567121 1157 ts_tablet_manager.cc:1918] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.567709 1165 raft_consensus.cc:357] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.567879 1165 raft_consensus.cc:383] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.567942 1165 raft_consensus.cc:738] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.568153 1165 consensus_queue.cc:260] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.569308 1165 ts_tablet_manager.cc:1428] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:54.569528 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet a8da9781ab32429297d0c875c1971501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.569597 1165 tablet_bootstrap.cc:492] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:54.570930 1165 tablet_bootstrap.cc:654] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.572831 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet a58d439254f8418091e3f23c8c274240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.581224 1157 log.cc:1198] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/e4c367a8ca184dfa9e4b11094ba76140
I20250109 20:52:54.587447 1157 ts_tablet_manager.cc:1939] T e4c367a8ca184dfa9e4b11094ba76140 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.589994 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet e4c367a8ca184dfa9e4b11094ba76140 (table default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]) successfully deleted
W20250109 20:52:54.591259 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9f5b6ca7ef84234b3d667d50ee5b8cd already in progress: creating tablet
W20250109 20:52:54.591513 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0f68bc0a564c4210bb29172a8759ebf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f68bc0a564c4210bb29172a8759ebf3 already in progress: creating tablet
W20250109 20:52:54.592413 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 89719a0d926c458d825371782980468c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89719a0d926c458d825371782980468c already in progress: creating tablet
W20250109 20:52:54.592603 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ffd3a1b1d3e44feda6dbaa8556e377fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffd3a1b1d3e44feda6dbaa8556e377fc already in progress: creating tablet
W20250109 20:52:54.592828 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4d2ad301c11943b69b59be18d6775513 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2ad301c11943b69b59be18d6775513 already in progress: creating tablet
I20250109 20:52:54.573453 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 86a59ce429b847f1b035b4dd06e39f70 (DEFAULT_TABLE table=default.Table_2ea95ef93df94b41a15c6f2ce7252567 [id=d55ef52b57c346acb6e5fea55de715fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.574314 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bd758fa2fa4cebbdf7faf3856f1941. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.610383 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bc408ae5174be89dbc2a13de464dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.563439 1806 raft_consensus.cc:513] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.610991 1806 leader_election.cc:290] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.611101 1158 raft_consensus.cc:738] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.611400 1806 raft_consensus.cc:491] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.611547 1806 raft_consensus.cc:513] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.611862 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3d0bfb39db40758e82be511d033b4c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.612048 1806 leader_election.cc:290] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.612772 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62caa775ef1241f68c0323a2d228c8cd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:54.614049 990 leader_election.cc:343] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:54.614982 990 leader_election.cc:343] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.615410 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24377e7b6354110a36711c69a878c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.617007 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86a59ce429b847f1b035b4dd06e39f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.617117 1158 consensus_queue.cc:260] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.579165 1166 ts_tablet_manager.cc:1939] T 2d96a569e6be4e01a26918d115cfbcf9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:54.620441 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1132c08b23ea4849a78b4809cbe44360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.620707 1158 ts_tablet_manager.cc:1428] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.349s user 0.023s sys 0.000s
I20250109 20:52:54.621122 1166 tablet_replica.cc:331] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:54.621384 1166 raft_consensus.cc:2238] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.621547 1166 raft_consensus.cc:2267] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.622931 1166 ts_tablet_manager.cc:1905] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.581231 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2ad301c11943b69b59be18d6775513 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b1822cd5744b5197d8141fc66539a0) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.623662 1158 tablet_bootstrap.cc:492] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:54.625396 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f948716bfa408193144275d7c9f5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.625573 1158 tablet_bootstrap.cc:654] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.628369 1166 ts_tablet_manager.cc:1918] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.628495 1166 log.cc:1198] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4d8e7081cac440389bec5dfac2391beb
I20250109 20:52:54.629705 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.581741 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 (DEFAULT_TABLE table=default.Table_6c6662f9bad8425db77969d4150d9c38 [id=50f790f5c8f243ba9c12c2a6bd515608]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.583469 1880 raft_consensus.cc:491] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.589143 1176 tablet_bootstrap.cc:492] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.597222 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025cdef634d4626a33f01c362a1388a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.599530 1026 tablet_service.cc:1467] Processing CreateTablet for tablet a6006590b7764efc901bf51ca4e19ab3 (DEFAULT_TABLE table=default.Table_cf33c8d126254dd580ef3fb97ecad07f [id=ab636e0f2f5d495c9a60d3a03dbd2d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.599885 1878 raft_consensus.cc:491] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.634706 1176 ts_tablet_manager.cc:1397] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.330s user 0.003s sys 0.008s
I20250109 20:52:54.634732 1880 raft_consensus.cc:513] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.634781 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6787f74100894fe6a7053a6d92ea7b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.637005 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deed945f2ecf4290ad002662c3f66030" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.639113 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ebf92383a5d419ba3e24ea4eea7ee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.639312 1878 raft_consensus.cc:513] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:54.640062 990 leader_election.cc:343] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.637549 1880 leader_election.cc:290] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.645927 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3997167fc6ee4cd0ae2401aa8924f684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.646329 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6006590b7764efc901bf51ca4e19ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.652299 1166 ts_tablet_manager.cc:1939] T 4d8e7081cac440389bec5dfac2391beb P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:54.658123 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95735385d5ab4fe0b29e33bad14eb95c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.658773 1158 tablet_bootstrap.cc:492] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:54.658931 1158 ts_tablet_manager.cc:1397] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250109 20:52:54.659732 1158 raft_consensus.cc:357] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.659984 1158 raft_consensus.cc:383] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.660043 1158 raft_consensus.cc:738] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.660281 1158 consensus_queue.cc:260] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:54.661232 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 192914us
W20250109 20:52:54.661610 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd because the tablet was not found. No further retry: Not found: Tablet not found: b9f5b6ca7ef84234b3d667d50ee5b8cd
W20250109 20:52:54.661959 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0f68bc0a564c4210bb29172a8759ebf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f68bc0a564c4210bb29172a8759ebf3 already in progress: creating tablet
I20250109 20:52:54.665405 885 tablet_service.cc:1514] Processing DeleteTablet for tablet dae574bf93d649099252e9e7b09b748c with delete_type TABLET_DATA_DELETED (replaced by tablet dc235614171f4a61b75492ddeb3f0fe6) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.665752 1160 tablet_replica.cc:331] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:54.665974 1160 raft_consensus.cc:2238] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.666115 1160 raft_consensus.cc:2267] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.666829 1160 ts_tablet_manager.cc:1905] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.668241 1158 ts_tablet_manager.cc:1428] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:52:54.669624 1158 tablet_bootstrap.cc:492] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:54.674018 1160 ts_tablet_manager.cc:1918] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.674117 1160 log.cc:1198] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/dae574bf93d649099252e9e7b09b748c
I20250109 20:52:54.674391 1160 ts_tablet_manager.cc:1939] T dae574bf93d649099252e9e7b09b748c P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:54.680099 1158 tablet_bootstrap.cc:654] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.690892 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a6630c808e354ecdb168332d5c3ea8da
I20250109 20:52:54.694008 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6461f153a5994412ba05c3b687963fd1
I20250109 20:52:54.697503 750 tablet_service.cc:1514] Processing DeleteTablet for tablet a6630c808e354ecdb168332d5c3ea8da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.697780 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6461f153a5994412ba05c3b687963fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:54.698376 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a6630c808e354ecdb168332d5c3ea8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6630c808e354ecdb168332d5c3ea8da already in progress: creating tablet
W20250109 20:52:54.699849 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6461f153a5994412ba05c3b687963fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6461f153a5994412ba05c3b687963fd1 already in progress: creating tablet
I20250109 20:52:54.701611 1158 tablet_bootstrap.cc:492] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:54.701802 1158 ts_tablet_manager.cc:1397] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.004s
I20250109 20:52:54.702409 1158 raft_consensus.cc:357] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.702620 1158 raft_consensus.cc:383] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.702692 1158 raft_consensus.cc:738] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.702934 1158 consensus_queue.cc:260] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.705447 1158 ts_tablet_manager.cc:1428] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250109 20:52:54.706373 1158 tablet_bootstrap.cc:492] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:54.707731 1158 tablet_bootstrap.cc:654] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.726146 1158 tablet_bootstrap.cc:492] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:54.726418 1158 ts_tablet_manager.cc:1397] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20250109 20:52:54.727324 1158 raft_consensus.cc:357] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.727646 1158 raft_consensus.cc:383] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.727762 1158 raft_consensus.cc:738] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.728091 1158 consensus_queue.cc:260] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.730113 1158 ts_tablet_manager.cc:1428] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:54.731333 1158 tablet_bootstrap.cc:492] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:54.733088 1158 tablet_bootstrap.cc:654] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.733352 1878 leader_election.cc:290] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:52:54.733698 990 leader_election.cc:343] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:54.733853 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 75179.5us
I20250109 20:52:54.735450 1176 raft_consensus.cc:357] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.736234 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a89a0b47d10414d9b6a9123af1f4ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.736475 1160 tablet_replica.cc:331] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:54.736652 1160 raft_consensus.cc:2238] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.736845 1160 raft_consensus.cc:2267] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.737619 1160 ts_tablet_manager.cc:1905] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.740758 885 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.741655 885 tablet_service.cc:1514] Processing DeleteTablet for tablet a6630c808e354ecdb168332d5c3ea8da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.742126 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6461f153a5994412ba05c3b687963fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.742477 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet dae574bf93d649099252e9e7b09b748c (table default.Table_8c82f4521cc94881ad01252c01dd81b8 [id=aea1bb0e692345379ba28d2c4c7f3fb8]) successfully deleted
I20250109 20:52:54.742785 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2ad301c11943b69b59be18d6775513 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b1822cd5744b5197d8141fc66539a0) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.743886 1160 ts_tablet_manager.cc:1918] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.743997 1160 log.cc:1198] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2a89a0b47d10414d9b6a9123af1f4ab3
I20250109 20:52:54.744306 1160 ts_tablet_manager.cc:1939] T 2a89a0b47d10414d9b6a9123af1f4ab3 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:54.749902 1158 tablet_bootstrap.cc:492] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:54.750029 1158 ts_tablet_manager.cc:1397] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20250109 20:52:54.750609 1158 raft_consensus.cc:357] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.750767 1158 raft_consensus.cc:383] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.750802 1158 raft_consensus.cc:738] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.750974 1158 consensus_queue.cc:260] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.752228 1158 ts_tablet_manager.cc:1428] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:54.753105 1158 tablet_bootstrap.cc:492] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:54.754786 1158 tablet_bootstrap.cc:654] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:54.758026 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2db4e85cd4094cf0b5b699402346b980 consensus state has no leader
I20250109 20:52:54.758767 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 31de9638669040db9674d3aa7df32103 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:54.764078 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 8539cefebb844b3098a308ca580b0e92 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:54.767355 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c5e9b31b27949c2948cb64b6124ee72 consensus state has no leader
I20250109 20:52:54.767596 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 2f5609eaf86d4e8ea3d3da41243c927f from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:54.769474 1878 raft_consensus.cc:491] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.770272 447 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
I20250109 20:52:54.771075 447 master_service.cc:946] ReplaceTablet: received request to replace tablet d6b06e8616df461d89c58d414fc0cfbd from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:54.772118 1176 raft_consensus.cc:383] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.772218 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5c79be4802d64c3f8e53093dba415e6e" } new_table_name: "default.Table_c8691eae9ae94278acb07e41ff9e1cfe" modify_external_catalogs: true
I20250109 20:52:54.772981 1810 raft_consensus.cc:491] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.773113 1810 raft_consensus.cc:513] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.773407 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
I20250109 20:52:54.773766 1810 leader_election.cc:290] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.774147 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_94c3bc135f0e4e028b93205980b55227"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:54.774362 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31b9785ee21418ba13b426ba94cfa9b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.774542 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8d504623c1894cb8843ead0ee687ea13" } modify_external_catalogs: true
I20250109 20:52:54.774725 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8d504623c1894cb8843ead0ee687ea13" } modify_external_catalogs: true
I20250109 20:52:54.775281 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_028d1b883d75407dbfa05dc1c7ba26cf" } modify_external_catalogs: true
I20250109 20:52:54.775461 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_028d1b883d75407dbfa05dc1c7ba26cf" } modify_external_catalogs: true
W20250109 20:52:54.779354 712 leader_election.cc:343] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.779758 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d6cafa4ebffe4a048881a673e64692d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:54.782137 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
I20250109 20:52:54.783679 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_534986b560ef4cbdaf8e21e045a76d29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:54.784375 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c918439b959a4a4fa4f2c75750ff33ca" } new_table_name: "default.Table_3816c8d68286465584634325363a453e" modify_external_catalogs: true
I20250109 20:52:54.789321 885 tablet_service.cc:1514] Processing DeleteTablet for tablet cba68160be6145ddbbd2019c9b5f0076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.790333 1762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:52:54.790584 1762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: LEADER
I20250109 20:52:54.791405 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b77c8d6adbf44375aff2782a83643255 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.791689 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8654c154dab465d8991910698027c3f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:54.792090 1878 raft_consensus.cc:513] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.792753 1878 leader_election.cc:290] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.793413 1054 raft_consensus.cc:1270] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:54.793712 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e615ce8e124cfe9432778b6b58e117" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:54.793926 1052 raft_consensus.cc:2463] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:54.794600 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4015c0e66dfd47bfb9855f82044f01e7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:54.795285 990 leader_election.cc:343] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.795514 1176 raft_consensus.cc:738] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.795565 1055 raft_consensus.cc:2463] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:54.795764 1176 consensus_queue.cc:260] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.796293 1176 ts_tablet_manager.cc:1428] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.161s user 0.002s sys 0.000s
I20250109 20:52:54.796597 1176 tablet_bootstrap.cc:492] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:54.797626 1884 raft_consensus.cc:491] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.797855 1884 raft_consensus.cc:513] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.798484 1176 tablet_bootstrap.cc:654] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.799181 1884 leader_election.cc:290] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.799880 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5757b98100804c688f67e7111db2b622" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.800098 778 raft_consensus.cc:2463] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:52:54.800596 990 leader_election.cc:304] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:54.800932 1878 raft_consensus.cc:2798] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:54.801020 1878 raft_consensus.cc:491] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:54.801122 1878 raft_consensus.cc:3054] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.803797 1878 raft_consensus.cc:513] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.804682 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5757b98100804c688f67e7111db2b622" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:52:54.804879 778 raft_consensus.cc:3054] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.805336 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31b9785ee21418ba13b426ba94cfa9b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:54.805542 1051 raft_consensus.cc:2463] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:54.806581 713 leader_election.cc:304] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.806598 1810 consensus_queue.cc:1035] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:54.807000 1678 raft_consensus.cc:2798] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:54.807067 1678 raft_consensus.cc:491] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:54.807132 1678 raft_consensus.cc:3054] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.807271 778 raft_consensus.cc:2463] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:52:54.807641 990 leader_election.cc:304] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:54.807863 1878 leader_election.cc:290] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.807994 1878 raft_consensus.cc:2798] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:54.808115 1878 raft_consensus.cc:695] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:52:54.808343 1878 consensus_queue.cc:237] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.810494 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.810691 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet a6630c808e354ecdb168332d5c3ea8da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.810886 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6461f153a5994412ba05c3b687963fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.813297 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deed945f2ecf4290ad002662c3f66030" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.813503 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3d0bfb39db40758e82be511d033b4c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.813750 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62caa775ef1241f68c0323a2d228c8cd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.813997 1166 tablet_replica.cc:331] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:54.814004 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95735385d5ab4fe0b29e33bad14eb95c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.814203 1166 raft_consensus.cc:2238] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.814244 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4015c0e66dfd47bfb9855f82044f01e7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.814376 1165 tablet_bootstrap.cc:492] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:54.814365 1166 raft_consensus.cc:2267] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.814483 1165 ts_tablet_manager.cc:1397] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.245s user 0.052s sys 0.099s
I20250109 20:52:54.814932 1166 ts_tablet_manager.cc:1905] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.815142 1165 raft_consensus.cc:357] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.815346 1165 raft_consensus.cc:383] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.815372 713 leader_election.cc:304] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.815424 1165 raft_consensus.cc:738] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.815640 1810 raft_consensus.cc:2798] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:54.815619 1165 consensus_queue.cc:260] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.815706 1810 raft_consensus.cc:491] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:54.815762 1810 raft_consensus.cc:3054] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.816040 1165 ts_tablet_manager.cc:1428] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:52:54.816246 1165 tablet_bootstrap.cc:492] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:54.817646 1165 tablet_bootstrap.cc:654] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.817821 1810 raft_consensus.cc:513] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.818379 1810 leader_election.cc:290] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.819743 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8654c154dab465d8991910698027c3f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:54.820904 1810 consensus_queue.cc:1035] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:54.820983 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5757b98100804c688f67e7111db2b622" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:54.821309 712 leader_election.cc:343] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.821720 1810 consensus_queue.cc:1035] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:54.822702 713 leader_election.cc:304] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.822880 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5757b98100804c688f67e7111db2b622" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:54.823134 1810 raft_consensus.cc:2798] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:54.823192 1810 raft_consensus.cc:491] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:54.823236 1810 raft_consensus.cc:3054] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.824290 1678 raft_consensus.cc:513] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.825204 1678 leader_election.cc:290] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.825224 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e615ce8e124cfe9432778b6b58e117" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:54.825614 712 leader_election.cc:343] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.825754 1810 raft_consensus.cc:513] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.826622 1810 leader_election.cc:290] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:54.826668 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31b9785ee21418ba13b426ba94cfa9b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:54.827023 712 leader_election.cc:343] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.827421 1887 raft_consensus.cc:491] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.827565 1160 tablet_replica.cc:331] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:54.827538 1887 raft_consensus.cc:513] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.827805 1160 raft_consensus.cc:2238] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.827965 1160 raft_consensus.cc:2267] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.828130 1887 leader_election.cc:290] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.828528 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4025cdef634d4626a33f01c362a1388a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:54.829082 990 leader_election.cc:343] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.829404 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4025cdef634d4626a33f01c362a1388a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.829864 1160 ts_tablet_manager.cc:1905] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:54.829787 989 leader_election.cc:343] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.830052 989 leader_election.cc:304] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.830669 1887 raft_consensus.cc:2743] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:54.831329 989 leader_election.cc:343] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.831573 989 leader_election.cc:304] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.832171 1887 raft_consensus.cc:2743] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.796281 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2d96a569e6be4e01a26918d115cfbcf9 (table default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]) successfully deleted
I20250109 20:52:54.834607 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d6b06e8616df461d89c58d414fc0cfbd
I20250109 20:52:54.835150 447 catalog_manager.cc:6596] ReplaceTablet: tablet d6b06e8616df461d89c58d414fc0cfbd deleted and replaced by tablet 3d7f5947674a4d6ea08cc52e72f905f0
I20250109 20:52:54.835355 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4d8e7081cac440389bec5dfac2391beb (table default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3 [id=e1afc4c2550f49f4b3c7b17ea2f09f60]) successfully deleted
I20250109 20:52:54.836522 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_19a1e320401641a383cf7c3fc20b4713 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 2 got 0
I20250109 20:52:54.836511 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b06e8616df461d89c58d414fc0cfbd with delete_type TABLET_DATA_DELETED (replaced by tablet 3d7f5947674a4d6ea08cc52e72f905f0) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.836748 1157 tablet_replica.cc:331] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.836858 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b06e8616df461d89c58d414fc0cfbd with delete_type TABLET_DATA_DELETED (replaced by tablet 3d7f5947674a4d6ea08cc52e72f905f0) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.836903 1157 raft_consensus.cc:2238] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
W20250109 20:52:54.836939 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6cafa4ebffe4a048881a673e64692d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.837195 1157 raft_consensus.cc:2267] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:54.837181 989 leader_election.cc:343] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.837280 989 leader_election.cc:304] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.837692 1157 ts_tablet_manager.cc:1905] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:54.838665 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94c3bc135f0e4e028b93205980b55227 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.839835 893 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b06e8616df461d89c58d414fc0cfbd with delete_type TABLET_DATA_DELETED (replaced by tablet 3d7f5947674a4d6ea08cc52e72f905f0) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.840633 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2a89a0b47d10414d9b6a9123af1f4ab3 (table default.Table_ef823ba5464d429db5aa6b8c3ccc1934 [id=70a6ff8c791e41038b082b9004740857]) successfully deleted
W20250109 20:52:54.840858 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ffd3a1b1d3e44feda6dbaa8556e377fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffd3a1b1d3e44feda6dbaa8556e377fc already in progress: creating tablet
W20250109 20:52:54.841059 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a6630c808e354ecdb168332d5c3ea8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6630c808e354ecdb168332d5c3ea8da already in progress: creating tablet
W20250109 20:52:54.841230 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6461f153a5994412ba05c3b687963fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6461f153a5994412ba05c3b687963fd1 already in progress: creating tablet
W20250109 20:52:54.841356 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4d2ad301c11943b69b59be18d6775513 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2ad301c11943b69b59be18d6775513 already in progress: creating tablet
I20250109 20:52:54.841576 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b1b97f9ea66f4c7c89289094e6d57533" } new_table_name: "default.Table_d4091d8d942a404cbea7396af4558981" modify_external_catalogs: true
I20250109 20:52:54.842039 1887 raft_consensus.cc:2743] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:54.842352 989 leader_election.cc:343] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.842423 989 leader_election.cc:304] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.842566 1887 raft_consensus.cc:2743] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:54.841629 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_534986b560ef4cbdaf8e21e045a76d29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:54.842839 989 leader_election.cc:343] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.842904 989 leader_election.cc:304] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.843027 1157 ts_tablet_manager.cc:1918] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:54.843112 1157 log.cc:1198] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/d6b06e8616df461d89c58d414fc0cfbd
I20250109 20:52:54.843429 1157 ts_tablet_manager.cc:1939] T d6b06e8616df461d89c58d414fc0cfbd P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.844417 1887 raft_consensus.cc:2743] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.844570 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet d6b06e8616df461d89c58d414fc0cfbd (table default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]) successfully deleted
W20250109 20:52:54.844708 989 leader_election.cc:343] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:54.848744 989 leader_election.cc:343] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:54.849000 989 leader_election.cc:343] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.849090 989 leader_election.cc:304] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.849272 1887 raft_consensus.cc:2743] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:54.850733 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86276d5c75be47e1bc5763ee532ad8ab consensus state has no leader
W20250109 20:52:54.851375 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d845bac2539e478083bf7fe6cbc0c717 consensus state has no leader
I20250109 20:52:54.852674 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_456a04c3836e48deaa5e0a3665b784d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.853288 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_456a04c3836e48deaa5e0a3665b784d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.854696 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_67d85bf32e4c4a9283bf48b7265d32cc" } modify_external_catalogs: true
I20250109 20:52:54.854903 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_67d85bf32e4c4a9283bf48b7265d32cc" } modify_external_catalogs: true
I20250109 20:52:54.858963 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8654c154dab465d8991910698027c3f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:54.859113 1055 raft_consensus.cc:3054] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.859545 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 29f1978ebf88470b80b8d4e38c6bc609 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:54.860832 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6787f74100894fe6a7053a6d92ea7b96 consensus state has no leader
I20250109 20:52:54.861608 1055 raft_consensus.cc:2463] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:54.863574 1176 tablet_bootstrap.cc:492] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:54.863736 1176 ts_tablet_manager.cc:1397] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.001s
I20250109 20:52:54.864462 1176 raft_consensus.cc:357] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.864635 1176 raft_consensus.cc:383] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.864691 1176 raft_consensus.cc:738] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.864898 1176 consensus_queue.cc:260] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.865526 1176 ts_tablet_manager.cc:1428] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:54.865767 1176 tablet_bootstrap.cc:492] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:54.879328 1887 raft_consensus.cc:491] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.879485 1887 raft_consensus.cc:513] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.880187 1887 leader_election.cc:290] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.880753 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8da9781ab32429297d0c875c1971501" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.881731 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e615ce8e124cfe9432778b6b58e117" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:54.881894 1054 raft_consensus.cc:3054] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.882493 1165 tablet_bootstrap.cc:492] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:54.882624 1165 ts_tablet_manager.cc:1397] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.020s
I20250109 20:52:54.882809 713 leader_election.cc:304] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.883163 1810 raft_consensus.cc:2798] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:54.883260 1810 raft_consensus.cc:695] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:54.883366 1165 raft_consensus.cc:357] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.883522 1165 raft_consensus.cc:383] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.883566 1165 raft_consensus.cc:738] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.883466 1810 consensus_queue.cc:237] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.883719 1165 consensus_queue.cc:260] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.884166 1165 ts_tablet_manager.cc:1428] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:52:54.884222 1054 raft_consensus.cc:2463] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:54.884495 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31b9785ee21418ba13b426ba94cfa9b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:54.884552 1165 tablet_bootstrap.cc:492] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:54.884622 1053 raft_consensus.cc:3054] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:54.886190 1165 tablet_bootstrap.cc:654] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.887035 1053 raft_consensus.cc:2463] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:54.887797 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb563bd6802473fafedd08bb10d15c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:54.890367 990 leader_election.cc:343] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.894855 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9ef276a3879c406abf2b9f0a326cde2a" } modify_external_catalogs: true
I20250109 20:52:54.895126 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9ef276a3879c406abf2b9f0a326cde2a" } modify_external_catalogs: true
I20250109 20:52:54.895910 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8da9781ab32429297d0c875c1971501" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.896049 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3ebf8106cfdb42bdbf56ec609f15305d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:54.897053 713 leader_election.cc:304] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.897343 1678 raft_consensus.cc:2798] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:54.897799 713 leader_election.cc:304] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.898128 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_aab45704dc304088b76a3c55953aaef4" } modify_external_catalogs: true
I20250109 20:52:54.898298 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_aab45704dc304088b76a3c55953aaef4" } modify_external_catalogs: true
W20250109 20:52:54.896672 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ebf8106cfdb42bdbf56ec609f15305d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.902096 1910 raft_consensus.cc:2798] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:54.902256 1910 raft_consensus.cc:695] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:54.902586 1166 ts_tablet_manager.cc:1918] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.902540 1910 consensus_queue.cc:237] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.904575 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 2608f48f5950466e9ab2d12ee87503cb (table default.Table_8a7392fff9c0401fb5c701ee92cf9c41 [id=17d12fac83d6463fb91fab6729a08139]). Expected version 1 got 0
I20250109 20:52:54.905293 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9af5cc3af3424aa3bd127450d55d6b36 (DEFAULT_TABLE table=default.Table_a3861f342059441c9d03405107eb97c5 [id=945ad26e463b4e7797328e80461661e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.905678 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af5cc3af3424aa3bd127450d55d6b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.908181 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_3585fc2fc449422880a7ebe625c22d09 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 1 got 0
I20250109 20:52:54.909404 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757b98100804c688f67e7111db2b622 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f8419d8cc7e475ea43dd263cae87aca) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.909579 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.909657 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.909739 752 tablet_service.cc:1514] Processing DeleteTablet for tablet deed945f2ecf4290ad002662c3f66030 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.909611 1157 tablet_replica.cc:331] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.910003 1157 raft_consensus.cc:2238] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.910130 1157 raft_consensus.cc:2267] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.910583 1157 ts_tablet_manager.cc:1905] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.915139 1160 ts_tablet_manager.cc:1918] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.916292 1157 ts_tablet_manager.cc:1918] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.916388 1157 log.cc:1198] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/5757b98100804c688f67e7111db2b622
I20250109 20:52:54.916651 1157 ts_tablet_manager.cc:1939] T 5757b98100804c688f67e7111db2b622 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.917785 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 5757b98100804c688f67e7111db2b622 (table default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]) successfully deleted
I20250109 20:52:54.918169 1157 tablet_replica.cc:331] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.918403 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bdce283727d1491ba0968a17c1c00d87
I20250109 20:52:54.918414 1157 raft_consensus.cc:2238] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.918557 1157 raft_consensus.cc:2267] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.919224 1157 ts_tablet_manager.cc:1905] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.919312 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 773f2c6e6d7f415ab1218f2365c880a2
I20250109 20:52:54.920310 752 tablet_service.cc:1514] Processing DeleteTablet for tablet bdce283727d1491ba0968a17c1c00d87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.920428 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 773f2c6e6d7f415ab1218f2365c880a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.920781 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 29f1978ebf88470b80b8d4e38c6bc609
I20250109 20:52:54.921175 455 catalog_manager.cc:6596] ReplaceTablet: tablet 29f1978ebf88470b80b8d4e38c6bc609 deleted and replaced by tablet 6b7f0230e20d45acaa48894fa41c1219
I20250109 20:52:54.920258 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet bdce283727d1491ba0968a17c1c00d87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.920387 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 773f2c6e6d7f415ab1218f2365c880a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.921515 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 219a4b508050490cb3f1ee418e0edbf8
I20250109 20:52:54.921605 1166 log.cc:1198] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/7307ebdb0c794451a6d91d23d69fddc6
I20250109 20:52:54.921923 1166 ts_tablet_manager.cc:1939] T 7307ebdb0c794451a6d91d23d69fddc6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:54.922408 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f1978ebf88470b80b8d4e38c6bc609 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b7f0230e20d45acaa48894fa41c1219) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.922567 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.922595 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f1978ebf88470b80b8d4e38c6bc609 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b7f0230e20d45acaa48894fa41c1219) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.922626 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.923161 1166 tablet_replica.cc:331] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:54.923346 1166 raft_consensus.cc:2238] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.923501 1166 raft_consensus.cc:2267] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.923520 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 7307ebdb0c794451a6d91d23d69fddc6 (table default.Table_2e81ac284d7740eb9e743bb21d11a625 [id=6f85f27d20024666b822558de484bc6e]) successfully deleted
I20250109 20:52:54.924018 1166 ts_tablet_manager.cc:1905] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.928179 752 tablet_service.cc:1467] Processing CreateTablet for tablet 9af5cc3af3424aa3bd127450d55d6b36 (DEFAULT_TABLE table=default.Table_a3861f342059441c9d03405107eb97c5 [id=945ad26e463b4e7797328e80461661e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.928504 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af5cc3af3424aa3bd127450d55d6b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.928646 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 9af5cc3af3424aa3bd127450d55d6b36 (DEFAULT_TABLE table=default.Table_a3861f342059441c9d03405107eb97c5 [id=945ad26e463b4e7797328e80461661e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.928956 1678 raft_consensus.cc:695] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:54.929018 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af5cc3af3424aa3bd127450d55d6b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.929215 1678 consensus_queue.cc:237] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:54.935299 1910 consensus_queue.cc:1035] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:54.921317 1176 tablet_bootstrap.cc:654] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.936372 1158 tablet_bootstrap.cc:492] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:54.936550 1158 ts_tablet_manager.cc:1397] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.183s user 0.041s sys 0.028s
I20250109 20:52:54.924726 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 773f2c6e6d7f415ab1218f2365c880a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.924891 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f1978ebf88470b80b8d4e38c6bc609 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b7f0230e20d45acaa48894fa41c1219) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.925007 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.925100 893 tablet_service.cc:1514] Processing DeleteTablet for tablet bdce283727d1491ba0968a17c1c00d87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.936893 1160 log.cc:1198] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/cba68160be6145ddbbd2019c9b5f0076
I20250109 20:52:54.937249 1160 ts_tablet_manager.cc:1939] T cba68160be6145ddbbd2019c9b5f0076 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:54.937505 1158 raft_consensus.cc:357] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.937717 1158 raft_consensus.cc:383] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.937841 1158 raft_consensus.cc:738] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.938228 1160 tablet_replica.cc:331] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:54.938192 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet cba68160be6145ddbbd2019c9b5f0076 (table default.Table_b70135c5a9b548538e791703e28af8f7 [id=f556fe2500bb47efba4056c36ca34823]) successfully deleted
I20250109 20:52:54.938149 1158 consensus_queue.cc:260] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.938591 1160 raft_consensus.cc:2238] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.938740 1160 raft_consensus.cc:2267] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.938866 1158 ts_tablet_manager.cc:1428] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:54.939363 1158 tablet_bootstrap.cc:492] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:54.939741 893 tablet_service.cc:1467] Processing CreateTablet for tablet d5e93f3b4c36485a92cfe28d5156b146 (DEFAULT_TABLE table=default.Table_98aa779b222a4fa1bb03d9c1d84dc2bf [id=a66b3d63bb4047cf978faeb9e563e2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.939875 1887 raft_consensus.cc:491] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.939790 752 tablet_service.cc:1467] Processing CreateTablet for tablet d5e93f3b4c36485a92cfe28d5156b146 (DEFAULT_TABLE table=default.Table_98aa779b222a4fa1bb03d9c1d84dc2bf [id=a66b3d63bb4047cf978faeb9e563e2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.940075 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e93f3b4c36485a92cfe28d5156b146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.939997 1887 raft_consensus.cc:513] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.940253 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e93f3b4c36485a92cfe28d5156b146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.940506 1887 leader_election.cc:290] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.940558 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4e680808eea0416bb05d5d61e0cc055d (DEFAULT_TABLE table=default.Table_0309fd73acdf416d94c30a93bc3e1cfe [id=8cec5184f32a411aae6fd5749adb5b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.940831 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e680808eea0416bb05d5d61e0cc055d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.945060 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41583
--webserver_interface=127.0.104.60
--webserver_port=46069
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:52:54.947953 1158 tablet_bootstrap.cc:654] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.948534 1908 raft_consensus.cc:491] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.948666 1908 raft_consensus.cc:513] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.949331 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58d439254f8418091e3f23c8c274240" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:54.949961 1157 ts_tablet_manager.cc:1918] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.950048 1157 log.cc:1198] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2b47656fe1ae4c9ba4138bcd37c0e6db
W20250109 20:52:54.950098 990 leader_election.cc:343] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.950309 1157 ts_tablet_manager.cc:1939] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.950887 1160 ts_tablet_manager.cc:1905] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.951396 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deed945f2ecf4290ad002662c3f66030" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:54.953156 989 leader_election.cc:343] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.953266 989 leader_election.cc:304] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.954408 1878 raft_consensus.cc:2743] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:54.958635 712 leader_election.cc:343] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.961346 455 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_19a1e320401641a383cf7c3fc20b4713 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 2 got 0
I20250109 20:52:54.961630 455 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
I20250109 20:52:54.962268 1158 tablet_bootstrap.cc:492] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:54.962641 1158 ts_tablet_manager.cc:1397] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20250109 20:52:54.963145 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.963359 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee with delete_type TABLET_DATA_DELETED (replaced by tablet d9f18f244a344272bba2bc085686b61d) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.963934 1158 raft_consensus.cc:357] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.964579 1158 raft_consensus.cc:383] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.964773 1158 raft_consensus.cc:738] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.965131 1158 consensus_queue.cc:260] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.965727 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 78800ff1dd964d8ba8862e3f4af69336
I20250109 20:52:54.966537 1878 raft_consensus.cc:491] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:54.966673 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 374df8c6d1424697a3b3922f777e9870
I20250109 20:52:54.966655 1878 raft_consensus.cc:513] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.967203 1878 leader_election.cc:290] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:54.958679 1160 ts_tablet_manager.cc:1918] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.967710 1160 log.cc:1198] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/b77c8d6adbf44375aff2782a83643255
I20250109 20:52:54.967617 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be44ffc9e371452f8d5f4174fbfa752a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:54.968132 990 leader_election.cc:343] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.968539 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be44ffc9e371452f8d5f4174fbfa752a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:54.968643 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 70859us
I20250109 20:52:54.968884 1160 ts_tablet_manager.cc:1939] T b77c8d6adbf44375aff2782a83643255 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:54.969125 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 374df8c6d1424697a3b3922f777e9870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.969318 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58d439254f8418091e3f23c8c274240" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:54.969774 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 78800ff1dd964d8ba8862e3f4af69336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:54.970330 1158 ts_tablet_manager.cc:1428] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250109 20:52:54.970222 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d5e93f3b4c36485a92cfe28d5156b146 (DEFAULT_TABLE table=default.Table_98aa779b222a4fa1bb03d9c1d84dc2bf [id=a66b3d63bb4047cf978faeb9e563e2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.970427 446 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_cb4c99bfeae9477485df9f52b78ea759 [id=0430588a51ed49ae86290eaab54f79dc]). Expected version 1 got 0
I20250109 20:52:54.970607 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e93f3b4c36485a92cfe28d5156b146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:54.971192 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet b77c8d6adbf44375aff2782a83643255 (table default.Table_a5bb2f6d26e3423e93672a1954c238a7 [id=43c66cf1f4554c79b35bd2689050a945]) successfully deleted
W20250109 20:52:54.971542 989 leader_election.cc:343] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.971642 989 leader_election.cc:304] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:54.971956 989 leader_election.cc:343] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:54.972039 989 leader_election.cc:304] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:54.972064 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 78800ff1dd964d8ba8862e3f4af69336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.972215 1887 raft_consensus.cc:2743] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.972271 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 374df8c6d1424697a3b3922f777e9870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:54.972340 1887 raft_consensus.cc:2743] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:54.972550 1158 tablet_bootstrap.cc:492] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:54.972587 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 78800ff1dd964d8ba8862e3f4af69336 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.972810 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 374df8c6d1424697a3b3922f777e9870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:54.974102 1158 tablet_bootstrap.cc:654] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.974357 1160 tablet_replica.cc:331] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:54.974777 1160 raft_consensus.cc:2238] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.975207 1160 raft_consensus.cc:2267] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.976099 1160 ts_tablet_manager.cc:1905] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.977453 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2b47656fe1ae4c9ba4138bcd37c0e6db (table default.Table_0c3c6e0557ae46a0a8854d04f3d303e0 [id=c24699f5a3e742578b56eb8bf1a4dc97]) successfully deleted
I20250109 20:52:54.977901 1157 tablet_replica.cc:331] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:54.979450 1157 raft_consensus.cc:2238] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:54.979602 1157 raft_consensus.cc:2267] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:54.980240 1157 ts_tablet_manager.cc:1905] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:54.983331 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fe23a0c1fdf142f5a7297a0f35955a50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.983947 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe23a0c1fdf142f5a7297a0f35955a50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.983398 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cede232c02904c21a0d46c338205fa14" } new_table_name: "default.Table_17aab5c8aa9a4999b2eebd66dcae9f46" modify_external_catalogs: true
I20250109 20:52:54.985142 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ff034ea8af80440fbe26a4c05c434f2c" } new_table_name: "default.Table_ddeb6e2ad0844a41b5aeaf86765e188e" modify_external_catalogs: true
I20250109 20:52:54.983803 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_33b6f13c94484781be39523332e10aed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.985828 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33b6f13c94484781be39523332e10aed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.986236 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d53cb6e3c5514b2e89f54a956be59a83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.986591 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d53cb6e3c5514b2e89f54a956be59a83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.982895 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dfc57ba40e274837bd12f0b1085b2eed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:54.990648 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfc57ba40e274837bd12f0b1085b2eed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:54.991082 1157 ts_tablet_manager.cc:1918] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:54.991173 1157 log.cc:1198] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/95735385d5ab4fe0b29e33bad14eb95c
I20250109 20:52:54.991451 1157 ts_tablet_manager.cc:1939] T 95735385d5ab4fe0b29e33bad14eb95c P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:54.992429 1158 tablet_bootstrap.cc:492] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:54.992532 1158 ts_tablet_manager.cc:1397] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250109 20:52:54.994170 1158 raft_consensus.cc:357] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.994354 1158 raft_consensus.cc:383] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:54.994401 1158 raft_consensus.cc:738] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:54.995354 1158 consensus_queue.cc:260] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:54.995834 1158 ts_tablet_manager.cc:1428] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:54.996059 1158 tablet_bootstrap.cc:492] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:54.997524 1158 tablet_bootstrap.cc:654] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:54.997445 890 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee970cc32dd483fa588ff287b7314e5 (DEFAULT_TABLE table=default.Table_5a36dfb8096e4cf79407414ba122d0e5 [id=0ffec69b106542b0a4445b498ac27dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:54.997824 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee970cc32dd483fa588ff287b7314e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.000558 1160 ts_tablet_manager.cc:1918] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:55.000654 1160 log.cc:1198] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/d6b06e8616df461d89c58d414fc0cfbd
I20250109 20:52:55.002108 1157 tablet_replica.cc:331] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:55.002257 1157 raft_consensus.cc:2238] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.002367 1157 raft_consensus.cc:2267] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.004060 1160 ts_tablet_manager.cc:1939] T d6b06e8616df461d89c58d414fc0cfbd P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:55.004628 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet d6b06e8616df461d89c58d414fc0cfbd (table default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]) successfully deleted
I20250109 20:52:55.005746 1160 tablet_replica.cc:331] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:55.005896 1160 raft_consensus.cc:2238] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.006031 1160 raft_consensus.cc:2267] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.007158 1926 tablet.cc:1721] T e6cc2c7b12f945ea8307a187e1553226 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:55.012007 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee970cc32dd483fa588ff287b7314e5 (DEFAULT_TABLE table=default.Table_5a36dfb8096e4cf79407414ba122d0e5 [id=0ffec69b106542b0a4445b498ac27dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.011963 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7202bebfc9474ba2b0ad805191200c6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.012362 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee970cc32dd483fa588ff287b7314e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.012542 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7202bebfc9474ba2b0ad805191200c6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.013379 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_19a1e320401641a383cf7c3fc20b4713 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 2 got 0
I20250109 20:52:55.014346 750 tablet_service.cc:1467] Processing CreateTablet for tablet 4e680808eea0416bb05d5d61e0cc055d (DEFAULT_TABLE table=default.Table_0309fd73acdf416d94c30a93bc3e1cfe [id=8cec5184f32a411aae6fd5749adb5b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.014771 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e680808eea0416bb05d5d61e0cc055d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.016948 1878 raft_consensus.cc:491] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.017064 1878 raft_consensus.cc:513] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.017647 1878 leader_election.cc:290] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:55.014348 752 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee970cc32dd483fa588ff287b7314e5 (DEFAULT_TABLE table=default.Table_5a36dfb8096e4cf79407414ba122d0e5 [id=0ffec69b106542b0a4445b498ac27dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.017961 1925 tablet.cc:1721] T e6cc2c7b12f945ea8307a187e1553226 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:55.018209 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee970cc32dd483fa588ff287b7314e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.018142 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b31bd4072941d5aaf8c43dada3ccc7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.018220 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b31bd4072941d5aaf8c43dada3ccc7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:55.018527 989 leader_election.cc:343] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:55.018672 990 leader_election.cc:343] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.018741 990 leader_election.cc:304] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.020862 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 95735385d5ab4fe0b29e33bad14eb95c (table default.Table_82246d90d1534a7e845106bde0e381f0 [id=a1ff467d633e4d999b85452ff6372219]) successfully deleted
I20250109 20:52:55.022574 1878 raft_consensus.cc:2743] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:55.028048 755 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8d8d771b56407d94fcb23d76796039 (DEFAULT_TABLE table=default.Table_d72a28b8d2b94712805f7299f0ccccf6 [id=7f1e6b9a475945339812a0e27b397d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.028388 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c8d8d771b56407d94fcb23d76796039. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.028391 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0d09d4894634825ad3aa7933bfb6010 consensus state has no leader
W20250109 20:52:55.028957 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be05de223c3946c6b8a4899cf696fe26 consensus state has no leader
I20250109 20:52:55.038372 893 tablet_service.cc:1467] Processing CreateTablet for tablet 0ccc75429f7441f1b186879bc5d12f92 (DEFAULT_TABLE table=default.Table_65bd13ddff1b46929045b98b3c549a18 [id=5a8754edfdcf49a4bf15a0e96ebbc805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.039103 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8d8d771b56407d94fcb23d76796039 (DEFAULT_TABLE table=default.Table_d72a28b8d2b94712805f7299f0ccccf6 [id=7f1e6b9a475945339812a0e27b397d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.039489 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c8d8d771b56407d94fcb23d76796039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.040912 1160 ts_tablet_manager.cc:1905] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.046680 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ccc75429f7441f1b186879bc5d12f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.048075 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_19a1e320401641a383cf7c3fc20b4713 [id=e0737f359bfa4154b4fb71617efc829f] alter complete (version 2)
I20250109 20:52:55.052364 1160 ts_tablet_manager.cc:1918] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.052498 1160 log.cc:1198] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/6fb563bd6802473fafedd08bb10d15c1
I20250109 20:52:55.053928 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_017eb9f961124f788b1a150416f6ccb3" } new_table_name: "default.Table_cc7c3bfc538c44f5a8f958176b1311c7" modify_external_catalogs: true
I20250109 20:52:55.053946 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 4e680808eea0416bb05d5d61e0cc055d (DEFAULT_TABLE table=default.Table_0309fd73acdf416d94c30a93bc3e1cfe [id=8cec5184f32a411aae6fd5749adb5b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.054486 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e680808eea0416bb05d5d61e0cc055d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.055378 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ff758d5ff9a64f8db3450cff1aea31db" } new_table_name: "default.Table_b66f03a4dc5241978f3ef4e00ebcd0a0" modify_external_catalogs: true
I20250109 20:52:55.059129 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0ccc75429f7441f1b186879bc5d12f92 (DEFAULT_TABLE table=default.Table_65bd13ddff1b46929045b98b3c549a18 [id=5a8754edfdcf49a4bf15a0e96ebbc805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.059441 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ccc75429f7441f1b186879bc5d12f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.062165 750 tablet_service.cc:1467] Processing CreateTablet for tablet 0ccc75429f7441f1b186879bc5d12f92 (DEFAULT_TABLE table=default.Table_65bd13ddff1b46929045b98b3c549a18 [id=5a8754edfdcf49a4bf15a0e96ebbc805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.062515 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ccc75429f7441f1b186879bc5d12f92. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.064677 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a45cd5ffe7040e385d8cdfb97ed011a consensus state has no leader
I20250109 20:52:55.065291 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_54a6148851094ffc9fb3923fb869a9e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.065614 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_655023e44afe42d597ce3e6cd3a1b9bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.065855 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54a6148851094ffc9fb3923fb869a9e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:55.066179 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_655023e44afe42d597ce3e6cd3a1b9bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.068383 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6015c9f96787455ba6e143685449c213" } new_table_name: "default.Table_c3c42837503b4d97a261c44ed755ba17" modify_external_catalogs: true
I20250109 20:52:55.069463 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8d8d771b56407d94fcb23d76796039 (DEFAULT_TABLE table=default.Table_d72a28b8d2b94712805f7299f0ccccf6 [id=7f1e6b9a475945339812a0e27b397d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.069901 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c8d8d771b56407d94fcb23d76796039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.073140 1160 ts_tablet_manager.cc:1939] T 6fb563bd6802473fafedd08bb10d15c1 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:55.074070 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 6fb563bd6802473fafedd08bb10d15c1 (table default.Table_d2ee28b720884b02a77177a6c829d85a [id=dccf417b39fc4d2d87812e340ecd842e]) successfully deleted
W20250109 20:52:55.074688 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 773f2c6e6d7f415ab1218f2365c880a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 773f2c6e6d7f415ab1218f2365c880a2 already in progress: creating tablet
W20250109 20:52:55.074918 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 29f1978ebf88470b80b8d4e38c6bc609 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f1978ebf88470b80b8d4e38c6bc609 already in progress: creating tablet
W20250109 20:52:55.075138 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 219a4b508050490cb3f1ee418e0edbf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 219a4b508050490cb3f1ee418e0edbf8 already in progress: creating tablet
W20250109 20:52:55.075338 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet bdce283727d1491ba0968a17c1c00d87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdce283727d1491ba0968a17c1c00d87 already in progress: creating tablet
I20250109 20:52:55.075394 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b339612e60ae46b48f88dc6d43a1a530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.075567 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 374df8c6d1424697a3b3922f777e9870 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 374df8c6d1424697a3b3922f777e9870 already in progress: creating tablet
W20250109 20:52:55.075748 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 78800ff1dd964d8ba8862e3f4af69336 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78800ff1dd964d8ba8862e3f4af69336 already in progress: creating tablet
W20250109 20:52:55.075857 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b339612e60ae46b48f88dc6d43a1a530 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.078512 889 tablet_service.cc:1467] Processing CreateTablet for tablet a43961d6eaec494385b2ecf67d39818b (DEFAULT_TABLE table=default.Table_50c2ff1060fd42b6a01b6b4f11cf3c14 [id=d7f8e560e7b54599aa5be9bdc31b2f86]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:55.079770 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ebf92383a5d419ba3e24ea4eea7ee72 consensus state has no leader
I20250109 20:52:55.080147 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_85420187c03540a3832414efe59f28d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.080570 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85420187c03540a3832414efe59f28d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.081516 1176 tablet_bootstrap.cc:492] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.081753 1176 ts_tablet_manager.cc:1397] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.216s user 0.003s sys 0.001s
I20250109 20:52:55.082862 1176 raft_consensus.cc:357] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.083375 1176 raft_consensus.cc:383] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.083528 1176 raft_consensus.cc:738] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.083963 1176 consensus_queue.cc:260] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.085279 1908 leader_election.cc:290] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.086426 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43961d6eaec494385b2ecf67d39818b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.088470 1157 ts_tablet_manager.cc:1905] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.089486 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deed945f2ecf4290ad002662c3f66030" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:55.089643 1051 raft_consensus.cc:2463] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:55.090330 713 leader_election.cc:304] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:55.090493 713 raft_consensus.cc:2681] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:52:55.094489 1176 ts_tablet_manager.cc:1428] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250109 20:52:55.094795 1176 tablet_bootstrap.cc:492] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.095593 1157 ts_tablet_manager.cc:1918] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.096213 1176 tablet_bootstrap.cc:654] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.096352 1157 log.cc:1198] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/deed945f2ecf4290ad002662c3f66030
I20250109 20:52:55.096704 1157 ts_tablet_manager.cc:1939] T deed945f2ecf4290ad002662c3f66030 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:52:55.097193 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f3c276fab6a4d068d64ea5d23c9a595 consensus state has no leader
I20250109 20:52:55.098284 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet deed945f2ecf4290ad002662c3f66030 (table default.Table_7c349e16f1b547c2b54536d168ff9e4a [id=339708d383f743e59fade9cec26192f7]) successfully deleted
W20250109 20:52:55.100411 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet bdce283727d1491ba0968a17c1c00d87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdce283727d1491ba0968a17c1c00d87 already in progress: creating tablet
I20250109 20:52:55.100878 1157 tablet_replica.cc:331] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:55.101125 1157 raft_consensus.cc:2238] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.101312 1157 raft_consensus.cc:2267] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.102383 453 catalog_manager.cc:5526] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } }
I20250109 20:52:55.102684 1157 ts_tablet_manager.cc:1905] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.102855 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:55.103104 453 catalog_manager.cc:5526] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } }
I20250109 20:52:55.103520 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 06e615ce8e124cfe9432778b6b58e117 (table default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]). Expected version 1 got 0
I20250109 20:52:55.103686 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:55.103874 453 catalog_manager.cc:5526] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } }
I20250109 20:52:55.104204 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_cbde6546bcd3407ebd6470f02d881bdd [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 4 got 0
I20250109 20:52:55.105384 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 2a4f047e8fa34ca79e78915c61a8ba12 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:55.106909 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 141787us
I20250109 20:52:55.107803 1165 tablet_bootstrap.cc:492] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:55.109200 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 8d1630e4438e4337ba026b41f630a480 (table default.Table_d7539ac3866842c996d4ced88e764877 [id=b78805ca41e647e1927d2564fb00b21f]). Expected version 3 got 0
I20250109 20:52:55.109404 752 tablet_service.cc:1467] Processing CreateTablet for tablet a43961d6eaec494385b2ecf67d39818b (DEFAULT_TABLE table=default.Table_50c2ff1060fd42b6a01b6b4f11cf3c14 [id=d7f8e560e7b54599aa5be9bdc31b2f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.107956 1165 ts_tablet_manager.cc:1397] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.223s user 0.018s sys 0.027s
I20250109 20:52:55.109776 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43961d6eaec494385b2ecf67d39818b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.110437 1165 raft_consensus.cc:357] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.110607 1165 raft_consensus.cc:383] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.110695 1165 raft_consensus.cc:738] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.110957 1165 consensus_queue.cc:260] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.113461 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2a4f047e8fa34ca79e78915c61a8ba12
I20250109 20:52:55.113888 450 catalog_manager.cc:6596] ReplaceTablet: tablet 2a4f047e8fa34ca79e78915c61a8ba12 deleted and replaced by tablet b6b23623677b4e088c14af925b4e07d4
I20250109 20:52:55.116159 1165 ts_tablet_manager.cc:1428] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:52:55.116379 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 with delete_type TABLET_DATA_DELETED (replaced by tablet b6b23623677b4e088c14af925b4e07d4) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.116591 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 with delete_type TABLET_DATA_DELETED (replaced by tablet b6b23623677b4e088c14af925b4e07d4) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.116928 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 with delete_type TABLET_DATA_DELETED (replaced by tablet b6b23623677b4e088c14af925b4e07d4) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.118765 889 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c367a8ca184dfa9e4b11094ba76140 with delete_type TABLET_DATA_DELETED (replaced by tablet 6619b3dce3ed4325a4871c3d40b42c9d) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.119282 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2a4f047e8fa34ca79e78915c61a8ba12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a4f047e8fa34ca79e78915c61a8ba12 already in progress: creating tablet
I20250109 20:52:55.119477 1165 tablet_bootstrap.cc:492] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:55.119998 1160 tablet_replica.cc:331] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:55.120558 1160 raft_consensus.cc:2238] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.120801 1160 raft_consensus.cc:2267] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.121397 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_06aabfaaf12e49bfa1196ae2f5850d21" } modify_external_catalogs: true
I20250109 20:52:55.121572 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_06aabfaaf12e49bfa1196ae2f5850d21" } modify_external_catalogs: true
I20250109 20:52:55.121392 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f34c9c364f74435e80caff9986005108"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.122666 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f34c9c364f74435e80caff9986005108 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.122644 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a82f87794b0d48f09f851336ce4e89d5" } modify_external_catalogs: true
I20250109 20:52:55.122828 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a82f87794b0d48f09f851336ce4e89d5" } modify_external_catalogs: true
I20250109 20:52:55.123014 1160 ts_tablet_manager.cc:1905] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.125821 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a5679436e31841a59474006e2efcc11f" } modify_external_catalogs: true
I20250109 20:52:55.125946 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a5679436e31841a59474006e2efcc11f" } modify_external_catalogs: true
I20250109 20:52:55.130649 1157 ts_tablet_manager.cc:1918] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.130753 1157 log.cc:1198] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/773f2c6e6d7f415ab1218f2365c880a2
I20250109 20:52:55.131053 1157 ts_tablet_manager.cc:1939] T 773f2c6e6d7f415ab1218f2365c880a2 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:55.131145 1017 tablet_service.cc:1467] Processing CreateTablet for tablet a43961d6eaec494385b2ecf67d39818b (DEFAULT_TABLE table=default.Table_50c2ff1060fd42b6a01b6b4f11cf3c14 [id=d7f8e560e7b54599aa5be9bdc31b2f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.131494 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43961d6eaec494385b2ecf67d39818b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.131493 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 8d1630e4438e4337ba026b41f630a480 (table default.Table_d7539ac3866842c996d4ced88e764877 [id=b78805ca41e647e1927d2564fb00b21f]). Expected version 3 got 0
I20250109 20:52:55.131999 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 773f2c6e6d7f415ab1218f2365c880a2 (table default.Table_67d85bf32e4c4a9283bf48b7265d32cc [id=f61e7aec1ea74c409648bee9f041ee0f]) successfully deleted
I20250109 20:52:55.132797 1165 tablet_bootstrap.cc:654] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:55.132812 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 29f1978ebf88470b80b8d4e38c6bc609 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f1978ebf88470b80b8d4e38c6bc609 already in progress: creating tablet
W20250109 20:52:55.133023 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 219a4b508050490cb3f1ee418e0edbf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 219a4b508050490cb3f1ee418e0edbf8 already in progress: creating tablet
W20250109 20:52:55.133198 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 78800ff1dd964d8ba8862e3f4af69336 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78800ff1dd964d8ba8862e3f4af69336 already in progress: creating tablet
W20250109 20:52:55.133378 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 374df8c6d1424697a3b3922f777e9870 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 374df8c6d1424697a3b3922f777e9870 already in progress: creating tablet
W20250109 20:52:55.133577 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2a4f047e8fa34ca79e78915c61a8ba12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a4f047e8fa34ca79e78915c61a8ba12 already in progress: creating tablet
I20250109 20:52:55.137233 1158 tablet_bootstrap.cc:492] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:55.137351 1158 ts_tablet_manager.cc:1397] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.141s user 0.005s sys 0.018s
I20250109 20:52:55.138127 1158 raft_consensus.cc:357] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.138336 1158 raft_consensus.cc:383] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.138396 1158 raft_consensus.cc:738] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.138554 1160 ts_tablet_manager.cc:1918] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.138651 1160 log.cc:1198] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/e4c367a8ca184dfa9e4b11094ba76140
I20250109 20:52:55.138942 1160 ts_tablet_manager.cc:1939] T e4c367a8ca184dfa9e4b11094ba76140 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:55.140367 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet e4c367a8ca184dfa9e4b11094ba76140 (table default.Table_7771b3421a9449929095a6e25dcbf470 [id=c6a5f3d877dc4165b4db160a774f6d3f]) successfully deleted
I20250109 20:52:55.140734 1158 consensus_queue.cc:260] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.144066 750 tablet_service.cc:1467] Processing CreateTablet for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 (DEFAULT_TABLE table=default.Table_44d9cfca7ff645bb8aad32946b74f286 [id=a802ce427a7d45b689bcd06b0aee1e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.144409 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f3c6c8764a4af1a9111e147c9c6ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.145296 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c03310d8c1ff4b3a82dd0ce35992a319
I20250109 20:52:55.145988 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 741d13db17e847039d3f9e25cafc6db0
I20250109 20:52:55.145973 889 tablet_service.cc:1514] Processing DeleteTablet for tablet c03310d8c1ff4b3a82dd0ce35992a319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.146476 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c03310d8c1ff4b3a82dd0ce35992a319 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c03310d8c1ff4b3a82dd0ce35992a319 already in progress: creating tablet
I20250109 20:52:55.146502 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 741d13db17e847039d3f9e25cafc6db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.146562 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3dfd19df44fb463aab5812702cbbbfe4
I20250109 20:52:55.147238 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 741d13db17e847039d3f9e25cafc6db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:55.147681 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 741d13db17e847039d3f9e25cafc6db0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 741d13db17e847039d3f9e25cafc6db0 already in progress: creating tablet
I20250109 20:52:55.148118 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 741d13db17e847039d3f9e25cafc6db0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.148497 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd19df44fb463aab5812702cbbbfe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.148561 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd19df44fb463aab5812702cbbbfe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.148664 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dfd19df44fb463aab5812702cbbbfe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.148784 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c03310d8c1ff4b3a82dd0ce35992a319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:55.149432 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3dfd19df44fb463aab5812702cbbbfe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dfd19df44fb463aab5812702cbbbfe4 already in progress: creating tablet
W20250109 20:52:55.150354 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 741d13db17e847039d3f9e25cafc6db0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 741d13db17e847039d3f9e25cafc6db0 already in progress: creating tablet
W20250109 20:52:55.150530 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3dfd19df44fb463aab5812702cbbbfe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dfd19df44fb463aab5812702cbbbfe4 already in progress: creating tablet
W20250109 20:52:55.150693 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c03310d8c1ff4b3a82dd0ce35992a319 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c03310d8c1ff4b3a82dd0ce35992a319 already in progress: creating tablet
I20250109 20:52:55.153092 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet c03310d8c1ff4b3a82dd0ce35992a319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.153506 893 tablet_service.cc:1467] Processing CreateTablet for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 (DEFAULT_TABLE table=default.Table_44d9cfca7ff645bb8aad32946b74f286 [id=a802ce427a7d45b689bcd06b0aee1e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.153883 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f3c6c8764a4af1a9111e147c9c6ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.165279 1158 ts_tablet_manager.cc:1428] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250109 20:52:55.166620 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.166828 1157 tablet_replica.cc:331] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:55.166986 1157 raft_consensus.cc:2238] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.167112 1157 raft_consensus.cc:2267] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.169122 1157 ts_tablet_manager.cc:1905] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.174659 889 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8d0bed80f742acbb882732207fc8ca (DEFAULT_TABLE table=default.Table_bbdaad91fc3541f79444f6adea0e9e4f [id=76735681f028432a88d43ab8b05ddaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.174767 755 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8d0bed80f742acbb882732207fc8ca (DEFAULT_TABLE table=default.Table_bbdaad91fc3541f79444f6adea0e9e4f [id=76735681f028432a88d43ab8b05ddaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.175066 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8d0bed80f742acbb882732207fc8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.175110 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8d0bed80f742acbb882732207fc8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.181196 1157 ts_tablet_manager.cc:1918] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.181242 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 (DEFAULT_TABLE table=default.Table_44d9cfca7ff645bb8aad32946b74f286 [id=a802ce427a7d45b689bcd06b0aee1e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.181442 1157 log.cc:1198] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2d96a569e6be4e01a26918d115cfbcf9
I20250109 20:52:55.181610 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5d462230734767bb010f520c86c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.181874 1157 ts_tablet_manager.cc:1939] T 2d96a569e6be4e01a26918d115cfbcf9 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:55.182391 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2d96a569e6be4e01a26918d115cfbcf9 (table default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]) successfully deleted
I20250109 20:52:55.182722 1157 tablet_replica.cc:331] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:55.182968 1157 raft_consensus.cc:2238] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.183166 1157 raft_consensus.cc:2267] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.183952 1157 ts_tablet_manager.cc:1905] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.185312 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9cbf7236e62e4c22a28e6ccb34f76495" } modify_external_catalogs: true
I20250109 20:52:55.185531 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9cbf7236e62e4c22a28e6ccb34f76495" } modify_external_catalogs: true
I20250109 20:52:55.186328 889 tablet_service.cc:1467] Processing CreateTablet for tablet d758bb29a9294e9481bb15ca4b4429b0 (DEFAULT_TABLE table=default.Table_00dd367ba48b4c8c93d19c4ec1d7017c [id=5a33d4a487de4473b1ec4a47005fe5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.186656 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d758bb29a9294e9481bb15ca4b4429b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.188316 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8d0bed80f742acbb882732207fc8ca (DEFAULT_TABLE table=default.Table_bbdaad91fc3541f79444f6adea0e9e4f [id=76735681f028432a88d43ab8b05ddaf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.188856 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_887acb86017941c4b8636c1008d835d9" } modify_external_catalogs: true
I20250109 20:52:55.188984 1158 tablet_bootstrap.cc:492] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:55.189129 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_887acb86017941c4b8636c1008d835d9" } modify_external_catalogs: true
I20250109 20:52:55.191123 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 7a1d5b0210134ce1a33b5e7603998593 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.191684 1158 tablet_bootstrap.cc:654] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.193029 1166 ts_tablet_manager.cc:1918] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.193141 1166 log.cc:1198] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2b47656fe1ae4c9ba4138bcd37c0e6db
I20250109 20:52:55.193440 1166 ts_tablet_manager.cc:1939] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.193464 455 master_service.cc:946] ReplaceTablet: received request to replace tablet cd596614521a4a92a71b781b162f70b6 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.194236 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 656) took 1103 ms. Trace:
I20250109 20:52:55.194304 1166 rpcz_store.cc:276] 0109 20:52:54.090351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.091143 (+ 792us) service_pool.cc:225] Handling call
0109 20:52:54.923124 (+831981us) ts_tablet_manager.cc:1145] Deleting tablet 2b47656fe1ae4c9ba4138bcd37c0e6db
0109 20:52:54.923132 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.056309 (+133177us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.192936 (+136627us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.194218 (+ 1282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":831860}
I20250109 20:52:55.194802 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2b47656fe1ae4c9ba4138bcd37c0e6db (table default.Table_0c3c6e0557ae46a0a8854d04f3d303e0 [id=c24699f5a3e742578b56eb8bf1a4dc97]) successfully deleted
I20250109 20:52:55.195082 1166 tablet_replica.cc:331] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.195250 1166 raft_consensus.cc:2238] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.195382 1166 raft_consensus.cc:2267] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.195859 1166 ts_tablet_manager.cc:1905] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.196537 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 18f85944f66543fe8dd634ad0da0728d
I20250109 20:52:55.197343 1157 ts_tablet_manager.cc:1918] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.197443 1157 log.cc:1198] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ea5d462230734767bb010f520c86c5ab
I20250109 20:52:55.197419 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f85944f66543fe8dd634ad0da0728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.198133 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 18f85944f66543fe8dd634ad0da0728d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18f85944f66543fe8dd634ad0da0728d already in progress: creating tablet
I20250109 20:52:55.198223 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f85944f66543fe8dd634ad0da0728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.200075 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8d0bed80f742acbb882732207fc8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.200579 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f85944f66543fe8dd634ad0da0728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.200975 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 59ebeffe516c4c239ae2ee9040276e53
I20250109 20:52:55.201592 1157 ts_tablet_manager.cc:1939] T ea5d462230734767bb010f520c86c5ab P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:55.201907 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.202464 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 59ebeffe516c4c239ae2ee9040276e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59ebeffe516c4c239ae2ee9040276e53 already in progress: creating tablet
I20250109 20:52:55.202646 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7a1d5b0210134ce1a33b5e7603998593
I20250109 20:52:55.202692 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.202765 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ea5d462230734767bb010f520c86c5ab (table default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]) successfully deleted
I20250109 20:52:55.203105 446 catalog_manager.cc:6596] ReplaceTablet: tablet 7a1d5b0210134ce1a33b5e7603998593 deleted and replaced by tablet 33a20cac880641d1aaa87b2472fd7978
I20250109 20:52:55.203269 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1d5b0210134ce1a33b5e7603998593 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a20cac880641d1aaa87b2472fd7978) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.203791 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7a1d5b0210134ce1a33b5e7603998593 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a1d5b0210134ce1a33b5e7603998593 already in progress: creating tablet
I20250109 20:52:55.204125 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1d5b0210134ce1a33b5e7603998593 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a20cac880641d1aaa87b2472fd7978) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.204577 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1d5b0210134ce1a33b5e7603998593 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a20cac880641d1aaa87b2472fd7978) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:55.204932 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 18f85944f66543fe8dd634ad0da0728d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18f85944f66543fe8dd634ad0da0728d already in progress: creating tablet
I20250109 20:52:55.205006 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f3c6c8764a4af1a9111e147c9c6ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.205051 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.206105 1158 tablet_bootstrap.cc:492] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:55.206264 1158 ts_tablet_manager.cc:1397] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20250109 20:52:55.207162 1158 raft_consensus.cc:357] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.207469 1158 raft_consensus.cc:383] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.207576 1158 raft_consensus.cc:738] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.207901 1158 consensus_queue.cc:260] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.211844 1158 ts_tablet_manager.cc:1428] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:55.213826 752 tablet_service.cc:1467] Processing CreateTablet for tablet d758bb29a9294e9481bb15ca4b4429b0 (DEFAULT_TABLE table=default.Table_00dd367ba48b4c8c93d19c4ec1d7017c [id=5a33d4a487de4473b1ec4a47005fe5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.214211 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d758bb29a9294e9481bb15ca4b4429b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.214341 1158 tablet_bootstrap.cc:492] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:55.214345 1016 tablet_service.cc:1467] Processing CreateTablet for tablet d758bb29a9294e9481bb15ca4b4429b0 (DEFAULT_TABLE table=default.Table_00dd367ba48b4c8c93d19c4ec1d7017c [id=5a33d4a487de4473b1ec4a47005fe5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.214680 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d758bb29a9294e9481bb15ca4b4429b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.215960 1158 tablet_bootstrap.cc:654] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.222543 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:55.226341 752 tablet_service.cc:1467] Processing CreateTablet for tablet 866f74d8bf534206b6d1642dbb42b603 (DEFAULT_TABLE table=default.Table_d8074d797bd94aaea619f6772e0c644c [id=026d9c2d7b404ed79fee8fe53888c68e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.226751 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866f74d8bf534206b6d1642dbb42b603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.226819 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 866f74d8bf534206b6d1642dbb42b603 (DEFAULT_TABLE table=default.Table_d8074d797bd94aaea619f6772e0c644c [id=026d9c2d7b404ed79fee8fe53888c68e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.227162 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866f74d8bf534206b6d1642dbb42b603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.228149 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 73e82535780c47ca8b9cd94b013b4b42 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:55.228539 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7a1d5b0210134ce1a33b5e7603998593 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a1d5b0210134ce1a33b5e7603998593 already in progress: creating tablet
W20250109 20:52:55.229511 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 59ebeffe516c4c239ae2ee9040276e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59ebeffe516c4c239ae2ee9040276e53 already in progress: creating tablet
I20250109 20:52:55.231948 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:55.234354 1051 raft_consensus.cc:1270] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250109 20:52:55.234540 712 consensus_peers.cc:487] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:55.234755 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_456a04c3836e48deaa5e0a3665b784d7" } new_table_name: "default.Table_a997c69220da46df9489620287286652" modify_external_catalogs: true
I20250109 20:52:55.235160 1917 consensus_queue.cc:1035] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:55.236668 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5e6c78d0dceb4466bb81f225c45bee0d" } new_table_name: "default.Table_422152ed650445aeb8f7af679f9c1d12" modify_external_catalogs: true
I20250109 20:52:55.239264 885 tablet_service.cc:1467] Processing CreateTablet for tablet 0224d985d9734052a3eef76eb829ea94 (DEFAULT_TABLE table=default.Table_acafc59f57e24e4e98ae6b7b3a50fccb [id=59d22c20d04a4aec8ee3949563971305]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.239591 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0224d985d9734052a3eef76eb829ea94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.240865 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_875ee088902c4843a6b6a6403666cd51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.241282 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_875ee088902c4843a6b6a6403666cd51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.242331 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7b61b7cfa1b14d94a0150b0590511425"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.242713 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b61b7cfa1b14d94a0150b0590511425 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:55.243860 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b107b4c26a94898b744654195bfe567 has no consensus state
I20250109 20:52:55.245853 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 95735385d5ab4fe0b29e33bad14eb95c from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.246448 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9fbed746bea74b4086ef2f2dc12674b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.246915 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fbed746bea74b4086ef2f2dc12674b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:55.247156 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50039da210df47ab8476e28387c8a802 consensus state has no leader
I20250109 20:52:55.247311 1165 tablet_bootstrap.cc:492] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:55.247439 1165 ts_tablet_manager.cc:1397] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.128s user 0.014s sys 0.030s
I20250109 20:52:55.248152 1165 raft_consensus.cc:357] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.248313 1165 raft_consensus.cc:383] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.248363 1165 raft_consensus.cc:738] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.248554 1165 consensus_queue.cc:260] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.248760 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_676e3bec2c46486b82b8e636ba06a4d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.249086 1165 ts_tablet_manager.cc:1428] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250109 20:52:55.249167 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_676e3bec2c46486b82b8e636ba06a4d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.249338 1165 tablet_bootstrap.cc:492] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:55.250578 1165 tablet_bootstrap.cc:654] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.251456 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dc6847928fe0476aae2e436f966abd99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.251866 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc6847928fe0476aae2e436f966abd99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.252604 1176 tablet_bootstrap.cc:492] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.252830 1176 ts_tablet_manager.cc:1397] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.158s user 0.001s sys 0.001s
I20250109 20:52:55.253582 1176 raft_consensus.cc:357] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.253803 1176 raft_consensus.cc:383] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.253891 1176 raft_consensus.cc:738] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.254133 1176 consensus_queue.cc:260] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.254451 897 tablet_service.cc:1467] Processing CreateTablet for tablet b10e70b6b88f438392a63a46123bdad0 (DEFAULT_TABLE table=default.Table_4a5f5d985f414ece9c3a1e4962abe0b7 [id=8d521d3e8fde4d9f93e27e1d33eb0429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.254736 750 tablet_service.cc:1467] Processing CreateTablet for tablet 0224d985d9734052a3eef76eb829ea94 (DEFAULT_TABLE table=default.Table_acafc59f57e24e4e98ae6b7b3a50fccb [id=59d22c20d04a4aec8ee3949563971305]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.255219 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10e70b6b88f438392a63a46123bdad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.255477 752 tablet_service.cc:1467] Processing CreateTablet for tablet b10e70b6b88f438392a63a46123bdad0 (DEFAULT_TABLE table=default.Table_4a5f5d985f414ece9c3a1e4962abe0b7 [id=8d521d3e8fde4d9f93e27e1d33eb0429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.256003 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10e70b6b88f438392a63a46123bdad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.256093 887 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3e7f598ca4696b733abd94238f85f (DEFAULT_TABLE table=default.Table_503a302a104946dba7674a2aba9dba11 [id=c699721e0e8340b1a72dc5cae071fba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.256361 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d3e7f598ca4696b733abd94238f85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.256842 1176 ts_tablet_manager.cc:1428] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:52:55.257669 1176 tablet_bootstrap.cc:492] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.258937 1176 tablet_bootstrap.cc:654] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.259286 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0224d985d9734052a3eef76eb829ea94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.262413 885 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcadec0c3184feaad4a883e5588f310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.262686 1160 tablet_replica.cc:331] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:55.262888 1160 raft_consensus.cc:2238] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.263031 1160 raft_consensus.cc:2267] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.266251 1160 ts_tablet_manager.cc:1905] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.269049 1016 tablet_service.cc:1467] Processing CreateTablet for tablet b10e70b6b88f438392a63a46123bdad0 (DEFAULT_TABLE table=default.Table_4a5f5d985f414ece9c3a1e4962abe0b7 [id=8d521d3e8fde4d9f93e27e1d33eb0429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.269471 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10e70b6b88f438392a63a46123bdad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.271910 1160 ts_tablet_manager.cc:1918] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.272053 1160 log.cc:1198] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/fbcadec0c3184feaad4a883e5588f310
I20250109 20:52:55.272372 1160 ts_tablet_manager.cc:1939] T fbcadec0c3184feaad4a883e5588f310 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:55.272960 889 tablet_service.cc:1467] Processing CreateTablet for tablet 866f74d8bf534206b6d1642dbb42b603 (DEFAULT_TABLE table=default.Table_d8074d797bd94aaea619f6772e0c644c [id=026d9c2d7b404ed79fee8fe53888c68e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.273319 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866f74d8bf534206b6d1642dbb42b603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.273399 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet fbcadec0c3184feaad4a883e5588f310 (table default.Table_e951676901de4adab1fb93a2d265da4f [id=97659ab8e1774179b7854d0985954b0b]) successfully deleted
I20250109 20:52:55.273442 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3a269f56a3b446c1973d3b9a24f0ec8f" } new_table_name: "default.Table_500eb283bc7f4131b62740a5e293b785" modify_external_catalogs: true
I20250109 20:52:55.274724 1176 tablet_bootstrap.cc:492] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.274868 1176 ts_tablet_manager.cc:1397] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.006s
W20250109 20:52:55.278743 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3974c4196d4043be9ad6e0452ed9cb7b consensus state has no leader
I20250109 20:52:55.279454 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 0224d985d9734052a3eef76eb829ea94 (DEFAULT_TABLE table=default.Table_acafc59f57e24e4e98ae6b7b3a50fccb [id=59d22c20d04a4aec8ee3949563971305]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.279832 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0224d985d9734052a3eef76eb829ea94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.280730 1176 raft_consensus.cc:357] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.281008 1176 raft_consensus.cc:383] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.281075 1176 raft_consensus.cc:738] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.281282 1176 consensus_queue.cc:260] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.282438 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7d9492ed2e5b44ada92f6fa5faa03ab9" } new_table_name: "default.Table_10b2d04e192c4740b1dffd0431e96880" modify_external_catalogs: true
I20250109 20:52:55.283492 1925 tablet.cc:1721] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:55.286978 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_cbde6546bcd3407ebd6470f02d881bdd [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 4 got 0
W20250109 20:52:55.288967 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 50039da210df47ab8476e28387c8a802 (table default.Table_422152ed650445aeb8f7af679f9c1d12 [id=b296e8173f88419a8a14ac395611b4bc], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 50039da210df47ab8476e28387c8a802 consensus state has no leader
I20250109 20:52:55.289408 750 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3e7f598ca4696b733abd94238f85f (DEFAULT_TABLE table=default.Table_503a302a104946dba7674a2aba9dba11 [id=c699721e0e8340b1a72dc5cae071fba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.289837 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d3e7f598ca4696b733abd94238f85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.297441 1910 raft_consensus.cc:491] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.297597 1910 raft_consensus.cc:513] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.298271 1910 leader_election.cc:290] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.299077 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2608f48f5950466e9ab2d12ee87503cb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.300662 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5e1c8b41b7ee4ac08260bfdefdc0555f" } new_table_name: "default.Table_4661e42d6cc949cd9e496a9aa3119d6f" modify_external_catalogs: true
W20250109 20:52:55.301880 712 leader_election.cc:343] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.302520 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2df8048c9d2f4c6e85562681c0ed0b94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.302959 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2df8048c9d2f4c6e85562681c0ed0b94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.303905 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_02714fd933d14554904584c58daa3159"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.304266 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02714fd933d14554904584c58daa3159 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.304567 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2608f48f5950466e9ab2d12ee87503cb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:55.304785 1051 raft_consensus.cc:2463] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:55.305378 713 leader_election.cc:304] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.305838 1910 raft_consensus.cc:2798] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:55.305912 1910 raft_consensus.cc:491] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:55.305975 1910 raft_consensus.cc:3054] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.306855 1926 tablet.cc:1721] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:55.306902 1033 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3e7f598ca4696b733abd94238f85f (DEFAULT_TABLE table=default.Table_503a302a104946dba7674a2aba9dba11 [id=c699721e0e8340b1a72dc5cae071fba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.307199 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d3e7f598ca4696b733abd94238f85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.308696 1910 raft_consensus.cc:513] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.309019 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1a0c835a44b44b92e6c3dc1aa44e3 (DEFAULT_TABLE table=default.Table_69bb01d4e8aa4563a727e44e5f6b3c20 [id=3250dce0ceb140878903269acf434f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.309336 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd1a0c835a44b44b92e6c3dc1aa44e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.310499 752 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1a0c835a44b44b92e6c3dc1aa44e3 (DEFAULT_TABLE table=default.Table_69bb01d4e8aa4563a727e44e5f6b3c20 [id=3250dce0ceb140878903269acf434f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.310796 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd1a0c835a44b44b92e6c3dc1aa44e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.311218 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2608f48f5950466e9ab2d12ee87503cb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:55.313256 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cc87fbc188284d3f9099de392fb15757"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.313438 755 tablet_service.cc:1467] Processing CreateTablet for tablet d79437d800f44c5a8222eea7dc2cd4e3 (DEFAULT_TABLE table=default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:55.313668 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc87fbc188284d3f9099de392fb15757 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.313712 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d79437d800f44c5a8222eea7dc2cd4e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.315006 712 leader_election.cc:343] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.316110 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a2537a166da14ae39e657d27dc555f71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.316509 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2537a166da14ae39e657d27dc555f71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.317200 1910 leader_election.cc:290] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.318835 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2608f48f5950466e9ab2d12ee87503cb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:55.319037 1051 raft_consensus.cc:3054] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:52:55.319278 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a279331730c7482580650c24365c4cba consensus state has no leader
I20250109 20:52:55.321161 1051 raft_consensus.cc:2463] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:55.328752 713 leader_election.cc:304] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.329635 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e4bfa188b545349f358adfd0bc714b with delete_type TABLET_DATA_DELETED (replaced by tablet a1ed00f7954143c9ba9bf65d8a8a68d2) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.330139 890 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1a0c835a44b44b92e6c3dc1aa44e3 (DEFAULT_TABLE table=default.Table_69bb01d4e8aa4563a727e44e5f6b3c20 [id=3250dce0ceb140878903269acf434f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.330577 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd1a0c835a44b44b92e6c3dc1aa44e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.333226 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a83e584f7694f5e9407449f73f9c907 consensus state has no leader
I20250109 20:52:55.333237 1910 raft_consensus.cc:2798] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:55.334954 1910 raft_consensus.cc:695] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:55.341325 1910 consensus_queue.cc:237] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.343936 885 tablet_service.cc:1467] Processing CreateTablet for tablet 07cbc61d839d46d2bae8235d6c92fbbd (DEFAULT_TABLE table=default.Table_7f2a351186d44db99378b0fc3678356f [id=52fa32728d1f4bd084598da1367b6e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.344967 897 tablet_service.cc:1467] Processing CreateTablet for tablet f948f9c3f77d498a9043382d6d3ba72e (DEFAULT_TABLE table=default.Table_43fc99f8a3c8473fa4c52881041e22a9 [id=78178bac079a444490edb7be6b8df401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.345857 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cbc61d839d46d2bae8235d6c92fbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.346061 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f948f9c3f77d498a9043382d6d3ba72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.347069 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_cbde6546bcd3407ebd6470f02d881bdd [id=4d18944cc25a4da6a21e3040370599e0] alter complete (version 4)
I20250109 20:52:55.345885 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 07cbc61d839d46d2bae8235d6c92fbbd (DEFAULT_TABLE table=default.Table_7f2a351186d44db99378b0fc3678356f [id=52fa32728d1f4bd084598da1367b6e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.349243 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cbc61d839d46d2bae8235d6c92fbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.349884 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c195088c8764cc2a94609872588c4a4" } modify_external_catalogs: true
I20250109 20:52:55.350054 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c195088c8764cc2a94609872588c4a4" } modify_external_catalogs: true
I20250109 20:52:55.350909 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f456c719d70549da87d54647fd2d02aa" } modify_external_catalogs: true
I20250109 20:52:55.351197 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f456c719d70549da87d54647fd2d02aa" } modify_external_catalogs: true
I20250109 20:52:55.351897 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8fdf081b7e1d4d55b7fc92577bfa1b3a" } modify_external_catalogs: true
I20250109 20:52:55.352085 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8fdf081b7e1d4d55b7fc92577bfa1b3a" } modify_external_catalogs: true
I20250109 20:52:55.344849 752 tablet_service.cc:1467] Processing CreateTablet for tablet 07cbc61d839d46d2bae8235d6c92fbbd (DEFAULT_TABLE table=default.Table_7f2a351186d44db99378b0fc3678356f [id=52fa32728d1f4bd084598da1367b6e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.354357 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cbc61d839d46d2bae8235d6c92fbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.356076 1017 tablet_service.cc:1467] Processing CreateTablet for tablet d79437d800f44c5a8222eea7dc2cd4e3 (DEFAULT_TABLE table=default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.356009 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_15d6655499224a0487b0c1f7feb8e3b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.356526 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d79437d800f44c5a8222eea7dc2cd4e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.356515 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15d6655499224a0487b0c1f7feb8e3b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.358685 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f1534c7240fc49f684af61aef783ee24
I20250109 20:52:55.360023 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f1534c7240fc49f684af61aef783ee24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.360301 897 tablet_service.cc:1514] Processing DeleteTablet for tablet f1534c7240fc49f684af61aef783ee24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.360895 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f1534c7240fc49f684af61aef783ee24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1534c7240fc49f684af61aef783ee24 already in progress: creating tablet
I20250109 20:52:55.361392 893 tablet_service.cc:1467] Processing CreateTablet for tablet d79437d800f44c5a8222eea7dc2cd4e3 (DEFAULT_TABLE table=default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.362895 1176 ts_tablet_manager.cc:1428] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.088s user 0.003s sys 0.000s
I20250109 20:52:55.363193 1176 tablet_bootstrap.cc:492] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.364380 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f1534c7240fc49f684af61aef783ee24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:55.365376 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f1534c7240fc49f684af61aef783ee24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1534c7240fc49f684af61aef783ee24 already in progress: creating tablet
I20250109 20:52:55.366959 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc with delete_type TABLET_DATA_DELETED (replaced by tablet 829bec4f66c24180afd5271057a7d0e8) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.370070 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d79437d800f44c5a8222eea7dc2cd4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.372426 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a792f8c86cd041deb4186424efd0e68a
I20250109 20:52:55.373162 897 tablet_service.cc:1514] Processing DeleteTablet for tablet a792f8c86cd041deb4186424efd0e68a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.375375 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a792f8c86cd041deb4186424efd0e68a because the tablet was not found. No further retry: Not found: Tablet not found: a792f8c86cd041deb4186424efd0e68a
I20250109 20:52:55.375715 750 tablet_service.cc:1514] Processing DeleteTablet for tablet a792f8c86cd041deb4186424efd0e68a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.376076 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet a792f8c86cd041deb4186424efd0e68a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:55.376259 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a792f8c86cd041deb4186424efd0e68a because the tablet was not found. No further retry: Not found: Tablet not found: a792f8c86cd041deb4186424efd0e68a
I20250109 20:52:55.377657 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet edc803f2936b4538a5e70ef34e5cc5d4
I20250109 20:52:55.378525 885 tablet_service.cc:1514] Processing DeleteTablet for tablet edc803f2936b4538a5e70ef34e5cc5d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.379034 751 tablet_service.cc:1514] Processing DeleteTablet for tablet edc803f2936b4538a5e70ef34e5cc5d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:55.379052 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet edc803f2936b4538a5e70ef34e5cc5d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edc803f2936b4538a5e70ef34e5cc5d4 already in progress: creating tablet
I20250109 20:52:55.379226 897 tablet_service.cc:1467] Processing CreateTablet for tablet d40b8ce042f349d599ce2a3bd2e4a91d (DEFAULT_TABLE table=default.Table_2768784b71f9461bbe4081deaaae9588 [id=dcef80bb6149463abb98ff6a0616cb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.379576 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40b8ce042f349d599ce2a3bd2e4a91d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.379914 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet edc803f2936b4538a5e70ef34e5cc5d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edc803f2936b4538a5e70ef34e5cc5d4 already in progress: creating tablet
I20250109 20:52:55.381013 755 tablet_service.cc:1467] Processing CreateTablet for tablet f948f9c3f77d498a9043382d6d3ba72e (DEFAULT_TABLE table=default.Table_43fc99f8a3c8473fa4c52881041e22a9 [id=78178bac079a444490edb7be6b8df401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.381410 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f948f9c3f77d498a9043382d6d3ba72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.385499 1176 tablet_bootstrap.cc:654] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.394665 1165 tablet_bootstrap.cc:492] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:55.394636 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet edc803f2936b4538a5e70ef34e5cc5d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.394927 1165 ts_tablet_manager.cc:1397] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.146s user 0.002s sys 0.001s
I20250109 20:52:55.395682 1165 raft_consensus.cc:357] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.396385 1176 tablet_bootstrap.cc:492] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.396493 1176 ts_tablet_manager.cc:1397] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20250109 20:52:55.398137 1176 raft_consensus.cc:357] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.398294 1176 raft_consensus.cc:383] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.398331 1176 raft_consensus.cc:738] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.398494 1176 consensus_queue.cc:260] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.399015 1165 raft_consensus.cc:383] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.399117 1165 raft_consensus.cc:738] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.399120 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_455cff8c7fb2449f908ca21e7da023af" } modify_external_catalogs: true
I20250109 20:52:55.399284 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_455cff8c7fb2449f908ca21e7da023af" } modify_external_catalogs: true
I20250109 20:52:55.399358 1165 consensus_queue.cc:260] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.399590 1176 ts_tablet_manager.cc:1428] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:55.399830 1176 tablet_bootstrap.cc:492] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.402787 451 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 06e615ce8e124cfe9432778b6b58e117 (table default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]). Expected version 1 got 0
I20250109 20:52:55.406860 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 56fac585b2904ab490a6882dd5252a9a
I20250109 20:52:55.407178 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2d8fde33d419436ca5839413c9b448da" } modify_external_catalogs: true
I20250109 20:52:55.407352 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2d8fde33d419436ca5839413c9b448da" } modify_external_catalogs: true
I20250109 20:52:55.407886 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 56fac585b2904ab490a6882dd5252a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:55.408623 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 56fac585b2904ab490a6882dd5252a9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56fac585b2904ab490a6882dd5252a9a already in progress: creating tablet
I20250109 20:52:55.408941 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 56fac585b2904ab490a6882dd5252a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.409255 1165 ts_tablet_manager.cc:1428] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250109 20:52:55.411455 448 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 06e615ce8e124cfe9432778b6b58e117 (table default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]). Expected version 1 got 0
I20250109 20:52:55.412810 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5e2b224f1d0b424ca2d4ee027d3acf5d
I20250109 20:52:55.413622 1176 tablet_bootstrap.cc:654] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.414379 450 catalog_manager.cc:5526] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:55.414891 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 2608f48f5950466e9ab2d12ee87503cb (table default.Table_8a7392fff9c0401fb5c701ee92cf9c41 [id=17d12fac83d6463fb91fab6729a08139]). Expected version 1 got 0
I20250109 20:52:55.418649 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 33c7eef84a0c46de990aee95ada1e48d from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.418792 1158 tablet_bootstrap.cc:492] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:55.418933 1158 ts_tablet_manager.cc:1397] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.205s user 0.036s sys 0.064s
I20250109 20:52:55.419670 1158 raft_consensus.cc:357] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.419852 1158 raft_consensus.cc:383] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.419931 1158 raft_consensus.cc:738] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.420336 1033 tablet_service.cc:1467] Processing CreateTablet for tablet f948f9c3f77d498a9043382d6d3ba72e (DEFAULT_TABLE table=default.Table_43fc99f8a3c8473fa4c52881041e22a9 [id=78178bac079a444490edb7be6b8df401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.420694 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f948f9c3f77d498a9043382d6d3ba72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.424645 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 56fac585b2904ab490a6882dd5252a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.424772 1158 consensus_queue.cc:260] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.425330 1158 ts_tablet_manager.cc:1428] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250109 20:52:55.425649 1158 tablet_bootstrap.cc:492] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:55.427066 1158 tablet_bootstrap.cc:654] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.428814 752 tablet_service.cc:1467] Processing CreateTablet for tablet d40b8ce042f349d599ce2a3bd2e4a91d (DEFAULT_TABLE table=default.Table_2768784b71f9461bbe4081deaaae9588 [id=dcef80bb6149463abb98ff6a0616cb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.428938 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d40b8ce042f349d599ce2a3bd2e4a91d (DEFAULT_TABLE table=default.Table_2768784b71f9461bbe4081deaaae9588 [id=dcef80bb6149463abb98ff6a0616cb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.429170 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40b8ce042f349d599ce2a3bd2e4a91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.429216 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40b8ce042f349d599ce2a3bd2e4a91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.434108 1166 ts_tablet_manager.cc:1918] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.434216 1166 log.cc:1198] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/deed945f2ecf4290ad002662c3f66030
I20250109 20:52:55.434551 1166 ts_tablet_manager.cc:1939] T deed945f2ecf4290ad002662c3f66030 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.435184 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 33c7eef84a0c46de990aee95ada1e48d
I20250109 20:52:55.435616 451 catalog_manager.cc:6596] ReplaceTablet: tablet 33c7eef84a0c46de990aee95ada1e48d deleted and replaced by tablet cbbdc5198b314511abfa78d8c8727e18
I20250109 20:52:55.436347 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c7eef84a0c46de990aee95ada1e48d with delete_type TABLET_DATA_DELETED (replaced by tablet cbbdc5198b314511abfa78d8c8727e18) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.436467 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c7eef84a0c46de990aee95ada1e48d with delete_type TABLET_DATA_DELETED (replaced by tablet cbbdc5198b314511abfa78d8c8727e18) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:55.436965 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 33c7eef84a0c46de990aee95ada1e48d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c7eef84a0c46de990aee95ada1e48d already in progress: creating tablet
I20250109 20:52:55.439850 750 tablet_service.cc:1467] Processing CreateTablet for tablet 646a6db049704d4e8169af36b15edbe0 (DEFAULT_TABLE table=default.Table_684bc1fac0764a80acc3c9be2f5df7ea [id=fb7f9ebca30d4657be434ff5ee878191]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.440227 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646a6db049704d4e8169af36b15edbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.440667 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8190cb332af34a398ae8970567562d0e (DEFAULT_TABLE table=default.Table_0a8d43084a0e45b48d75867cfcbe83e1 [id=ab1a8574297540baaba0093c45ea9f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.440984 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8190cb332af34a398ae8970567562d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.440940 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8190cb332af34a398ae8970567562d0e (DEFAULT_TABLE table=default.Table_0a8d43084a0e45b48d75867cfcbe83e1 [id=ab1a8574297540baaba0093c45ea9f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.446139 1878 raft_consensus.cc:491] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.446275 1878 raft_consensus.cc:513] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.446838 1878 leader_election.cc:290] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:55.447007 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4cf7db6076db4bc4a5b7527f7b0a41cc" } new_table_name: "default.Table_49ecdce4b5a94e4a92d868acdaefb783" modify_external_catalogs: true
I20250109 20:52:55.447114 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe8a9e5b5fe406b988126cd1859eabc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.447249 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 646a6db049704d4e8169af36b15edbe0 (DEFAULT_TABLE table=default.Table_684bc1fac0764a80acc3c9be2f5df7ea [id=fb7f9ebca30d4657be434ff5ee878191]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.447537 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646a6db049704d4e8169af36b15edbe0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.447572 989 leader_election.cc:343] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.448585 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bc52c54bc34a44d88aab13982fc4e59b" } new_table_name: "default.Table_7e2171ec1bd442eb911f34154fbc9f58" modify_external_catalogs: true
W20250109 20:52:55.454169 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a85c36689d8f4b16b85db5ad3c84cac0 consensus state has no leader
I20250109 20:52:55.454564 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 658) took 1364 ms. Trace:
I20250109 20:52:55.454644 1166 rpcz_store.cc:276] 0109 20:52:54.090489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.091291 (+ 802us) service_pool.cc:225] Handling call
0109 20:52:55.195035 (+1103744us) ts_tablet_manager.cc:1145] Deleting tablet deed945f2ecf4290ad002662c3f66030
0109 20:52:55.195047 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.339618 (+144571us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.434036 (+ 94418us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.454545 (+ 20509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103609}
I20250109 20:52:55.454941 1166 tablet_replica.cc:331] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.455117 1166 raft_consensus.cc:2238] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.455124 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet deed945f2ecf4290ad002662c3f66030 (table default.Table_7c349e16f1b547c2b54536d168ff9e4a [id=339708d383f743e59fade9cec26192f7]) successfully deleted
I20250109 20:52:55.455247 1166 raft_consensus.cc:2267] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.455715 1166 ts_tablet_manager.cc:1905] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:55.464751 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bcd3219734314ae1ad03968fdf9c9cd0 consensus state has no leader
W20250109 20:52:55.466733 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 56fac585b2904ab490a6882dd5252a9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56fac585b2904ab490a6882dd5252a9a already in progress: creating tablet
I20250109 20:52:55.466943 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_54e2dbe4b2ac420ba5f1ebd060213e74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.468302 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c7eef84a0c46de990aee95ada1e48d with delete_type TABLET_DATA_DELETED (replaced by tablet cbbdc5198b314511abfa78d8c8727e18) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.468286 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_aff5b59bbc5445d8b655981c6f4ca537"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.468819 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aff5b59bbc5445d8b655981c6f4ca537 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.468969 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8190cb332af34a398ae8970567562d0e (DEFAULT_TABLE table=default.Table_0a8d43084a0e45b48d75867cfcbe83e1 [id=ab1a8574297540baaba0093c45ea9f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.469311 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8190cb332af34a398ae8970567562d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.472784 1166 ts_tablet_manager.cc:1918] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.472894 1166 log.cc:1198] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/95735385d5ab4fe0b29e33bad14eb95c
I20250109 20:52:55.472996 894 tablet_service.cc:1467] Processing CreateTablet for tablet f4f6196d820f4aecaff2701aaf56faf5 (DEFAULT_TABLE table=default.Table_77c9fc51089f43c0a1c2db8a4d77573f [id=0f3291f9fb3a4ac583055a2fb53980a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.473196 1166 ts_tablet_manager.cc:1939] T 95735385d5ab4fe0b29e33bad14eb95c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.473304 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f6196d820f4aecaff2701aaf56faf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.473865 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 659) took 1383 ms. Trace:
I20250109 20:52:55.474282 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_51c7449051f04e25bb0ee8eb814de9e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.474764 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51c7449051f04e25bb0ee8eb814de9e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.475941 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0be3c25688ad443cadd4f1c2c068f2c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.476333 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0be3c25688ad443cadd4f1c2c068f2c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.468676 887 tablet_service.cc:1467] Processing CreateTablet for tablet 646a6db049704d4e8169af36b15edbe0 (DEFAULT_TABLE table=default.Table_684bc1fac0764a80acc3c9be2f5df7ea [id=fb7f9ebca30d4657be434ff5ee878191]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.477535 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646a6db049704d4e8169af36b15edbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.478024 1026 tablet_service.cc:1467] Processing CreateTablet for tablet a792f8c86cd041deb4186424efd0e68a (DEFAULT_TABLE table=default.Table_8fdf081b7e1d4d55b7fc92577bfa1b3a [id=3dc32b9bc767455aba6f8dc97d7034ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.478327 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a792f8c86cd041deb4186424efd0e68a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.481284 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 33c7eef84a0c46de990aee95ada1e48d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c7eef84a0c46de990aee95ada1e48d already in progress: creating tablet
I20250109 20:52:55.481276 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8c1d631caa444686bedde93287738a48" } new_table_name: "default.Table_af24f0c294ec48e7ba9e760a802a7696" modify_external_catalogs: true
I20250109 20:52:55.481699 896 tablet_service.cc:1467] Processing CreateTablet for tablet 062f1d04adc14599ace583896a3117e0 (DEFAULT_TABLE table=default.Table_84dfe41ea9994c1aa62b04e2ec77e4c9 [id=5df146fe156743f0a506e95469ae9964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.481989 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062f1d04adc14599ace583896a3117e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.482327 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 062f1d04adc14599ace583896a3117e0 (DEFAULT_TABLE table=default.Table_84dfe41ea9994c1aa62b04e2ec77e4c9 [id=5df146fe156743f0a506e95469ae9964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.482599 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062f1d04adc14599ace583896a3117e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.482944 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cf48fa3edeeb4675a6e04b039bdee9ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.483327 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf48fa3edeeb4675a6e04b039bdee9ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.473932 1166 rpcz_store.cc:276] 0109 20:52:54.090549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.091446 (+ 897us) service_pool.cc:225] Handling call
0109 20:52:55.454901 (+1363455us) ts_tablet_manager.cc:1145] Deleting tablet 95735385d5ab4fe0b29e33bad14eb95c
0109 20:52:55.454910 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.458126 (+ 3216us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.460673 (+ 2547us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.473847 (+ 13174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363320}
I20250109 20:52:55.487164 1166 tablet_replica.cc:331] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.487263 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 95735385d5ab4fe0b29e33bad14eb95c (table default.Table_82246d90d1534a7e845106bde0e381f0 [id=a1ff467d633e4d999b85452ff6372219]) successfully deleted
I20250109 20:52:55.487282 885 tablet_service.cc:1467] Processing CreateTablet for tablet a792f8c86cd041deb4186424efd0e68a (DEFAULT_TABLE table=default.Table_8fdf081b7e1d4d55b7fc92577bfa1b3a [id=3dc32b9bc767455aba6f8dc97d7034ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.487334 1166 raft_consensus.cc:2238] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.487458 1166 raft_consensus.cc:2267] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.487561 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a792f8c86cd041deb4186424efd0e68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.487895 1166 ts_tablet_manager.cc:1905] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.488716 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 11dbbc9ceb1a4252b1605560bdb9b62a (DEFAULT_TABLE table=default.Table_532e9f42a7084d3c9037828c563a3f7b [id=b520db9fb09a4f1180e081161f6762ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.489046 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11dbbc9ceb1a4252b1605560bdb9b62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.490273 1165 tablet_bootstrap.cc:492] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:55.491559 1165 tablet_bootstrap.cc:654] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:55.492985 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 617029392e994f44a60c71ba2e340c57 consensus state has no leader
I20250109 20:52:55.498720 1017 tablet_service.cc:1467] Processing CreateTablet for tablet f4f6196d820f4aecaff2701aaf56faf5 (DEFAULT_TABLE table=default.Table_77c9fc51089f43c0a1c2db8a4d77573f [id=0f3291f9fb3a4ac583055a2fb53980a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.499157 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f6196d820f4aecaff2701aaf56faf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.501004 1166 ts_tablet_manager.cc:1918] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.501169 1166 log.cc:1198] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ea5d462230734767bb010f520c86c5ab
I20250109 20:52:55.501555 1166 ts_tablet_manager.cc:1939] T ea5d462230734767bb010f520c86c5ab P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:55.502094 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54e2dbe4b2ac420ba5f1ebd060213e74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.502285 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 650) took 1412 ms. Trace:
I20250109 20:52:55.502408 1166 rpcz_store.cc:276] 0109 20:52:54.089407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.090788 (+ 1381us) service_pool.cc:225] Handling call
0109 20:52:55.487124 (+1396336us) ts_tablet_manager.cc:1145] Deleting tablet ea5d462230734767bb010f520c86c5ab
0109 20:52:55.487134 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.490549 (+ 3415us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.500914 (+ 10365us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.502267 (+ 1353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395459}
I20250109 20:52:55.502801 1166 tablet_replica.cc:331] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.502959 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ea5d462230734767bb010f520c86c5ab (table default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]) successfully deleted
I20250109 20:52:55.503039 1166 raft_consensus.cc:2238] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:55.504293 1166 pending_rounds.cc:62] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Trying to abort 1 pending ops.
I20250109 20:52:55.507720 1166 pending_rounds.cc:69] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523672817721344 op_type: NO_OP noop_request { }
I20250109 20:52:55.507903 1166 raft_consensus.cc:2883] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:55.508090 1166 raft_consensus.cc:2267] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.509164 1166 ts_tablet_manager.cc:1905] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.512116 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 6787f74100894fe6a7053a6d92ea7b96 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.512646 896 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5f945b345d4c029798756e63867156 (DEFAULT_TABLE table=default.Table_3b054fb299c84cf49c242e3ccb44daf8 [id=a8d9617e1d1d4488abe06c7065bfee48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.513109 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5f945b345d4c029798756e63867156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.516103 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 55bf7303e7354aa99a7b5bb489f442fe (DEFAULT_TABLE table=default.Table_881894033dec46b3862c1185df595c40 [id=a3ccfed03acd4b2d9fe9bcef1683dc4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.516570 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 1591374d521a47ebb405a812fe656b19 (DEFAULT_TABLE table=default.Table_814844402c5e4defbb723b986171a2a2 [id=a4138fd6ff5246d582fc0d75b890c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.516758 887 tablet_service.cc:1467] Processing CreateTablet for tablet 1591374d521a47ebb405a812fe656b19 (DEFAULT_TABLE table=default.Table_814844402c5e4defbb723b986171a2a2 [id=a4138fd6ff5246d582fc0d75b890c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.517032 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6e020eeb03e041ed977779a07de5bb80 (DEFAULT_TABLE table=default.Table_3873238920c845cb8338b7a36ef75274 [id=8bec14b8b4b74d9c87fe86950f5128e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.517138 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1591374d521a47ebb405a812fe656b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.518615 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6787f74100894fe6a7053a6d92ea7b96
I20250109 20:52:55.519029 450 catalog_manager.cc:6596] ReplaceTablet: tablet 6787f74100894fe6a7053a6d92ea7b96 deleted and replaced by tablet d8bc15fecd8542e49ff5a0eb22211ec9
I20250109 20:52:55.519366 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6787f74100894fe6a7053a6d92ea7b96 with delete_type TABLET_DATA_DELETED (replaced by tablet d8bc15fecd8542e49ff5a0eb22211ec9) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.521629 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_69a70cbdfe8148838135d7f7da4e6619" } new_table_name: "default.Table_6ad624c51a204e2bbb07ab9b0897bd0c" modify_external_catalogs: true
W20250109 20:52:55.527202 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c4ffd6adbaf4f7083e7ba1a518ed813 consensus state has no leader
I20250109 20:52:55.528635 897 tablet_service.cc:1467] Processing CreateTablet for tablet 11dbbc9ceb1a4252b1605560bdb9b62a (DEFAULT_TABLE table=default.Table_532e9f42a7084d3c9037828c563a3f7b [id=b520db9fb09a4f1180e081161f6762ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.528851 880 tablet_service.cc:1467] Processing CreateTablet for tablet 55bf7303e7354aa99a7b5bb489f442fe (DEFAULT_TABLE table=default.Table_881894033dec46b3862c1185df595c40 [id=a3ccfed03acd4b2d9fe9bcef1683dc4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.528936 1166 ts_tablet_manager.cc:1918] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:55.529023 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11dbbc9ceb1a4252b1605560bdb9b62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.529137 1166 log.cc:1198] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/5757b98100804c688f67e7111db2b622
I20250109 20:52:55.529192 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55bf7303e7354aa99a7b5bb489f442fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.529218 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55bf7303e7354aa99a7b5bb489f442fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.529500 1166 ts_tablet_manager.cc:1939] T 5757b98100804c688f67e7111db2b622 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.532680 1176 tablet_bootstrap.cc:492] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.532830 1176 ts_tablet_manager.cc:1397] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.133s user 0.003s sys 0.004s
I20250109 20:52:55.533124 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6e020eeb03e041ed977779a07de5bb80 (DEFAULT_TABLE table=default.Table_3873238920c845cb8338b7a36ef75274 [id=8bec14b8b4b74d9c87fe86950f5128e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.533484 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e020eeb03e041ed977779a07de5bb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.533434 1176 raft_consensus.cc:357] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.533619 1176 raft_consensus.cc:383] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.533670 1176 raft_consensus.cc:738] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.533905 1176 consensus_queue.cc:260] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.535604 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6787f74100894fe6a7053a6d92ea7b96 with delete_type TABLET_DATA_DELETED (replaced by tablet d8bc15fecd8542e49ff5a0eb22211ec9) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.538203 1176 ts_tablet_manager.cc:1428] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250109 20:52:55.538479 1176 tablet_bootstrap.cc:492] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.538470 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5b5c1f8eb7b14686abcfdab2e28fa57d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.538975 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b5c1f8eb7b14686abcfdab2e28fa57d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:55.539180 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6787f74100894fe6a7053a6d92ea7b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6787f74100894fe6a7053a6d92ea7b96 already in progress: creating tablet
I20250109 20:52:55.539359 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_83f4a695b41741509248430f3d916085"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.539791 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83f4a695b41741509248430f3d916085 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.539880 1176 tablet_bootstrap.cc:654] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.540277 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cbde6546bcd3407ebd6470f02d881bdd" } new_table_name: "default.Table_7e98b984f52d4023ac3365b3e23f26e2" modify_external_catalogs: true
I20250109 20:52:55.540684 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1591374d521a47ebb405a812fe656b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.541142 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_642b497e51524120a0696f61734c20fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.541716 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_642b497e51524120a0696f61734c20fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.543071 1823 raft_consensus.cc:491] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.543192 1823 raft_consensus.cc:513] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.543701 1823 leader_election.cc:290] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.543926 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 653) took 1453 ms. Trace:
I20250109 20:52:55.543994 1166 rpcz_store.cc:276] 0109 20:52:54.090151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.090990 (+ 839us) service_pool.cc:225] Handling call
0109 20:52:55.502763 (+1411773us) ts_tablet_manager.cc:1145] Deleting tablet 5757b98100804c688f67e7111db2b622
0109 20:52:55.502771 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.511748 (+ 8977us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.528850 (+ 17102us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.543910 (+ 15060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411049}
I20250109 20:52:55.544245 1166 tablet_replica.cc:331] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.544328 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc2c7b12f945ea8307a187e1553226" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:55.544423 1166 raft_consensus.cc:2238] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.544467 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet af8c0bf27e994e6aa4cbedca16febe12 (table default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]). Expected version 1 got 0
I20250109 20:52:55.544538 1166 raft_consensus.cc:2267] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.545032 1166 ts_tablet_manager.cc:1905] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.546316 1165 tablet_bootstrap.cc:492] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:55.546486 1165 ts_tablet_manager.cc:1397] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.006s
I20250109 20:52:55.547128 1165 raft_consensus.cc:357] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.547341 1165 raft_consensus.cc:383] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.547431 1165 raft_consensus.cc:738] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.547673 1165 consensus_queue.cc:260] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.548177 1165 ts_tablet_manager.cc:1428] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:55.548920 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 5757b98100804c688f67e7111db2b622 (table default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]) successfully deleted
I20250109 20:52:55.550094 1165 tablet_bootstrap.cc:492] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:55.551528 1165 tablet_bootstrap.cc:654] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.552479 880 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.555065 1160 tablet_replica.cc:331] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:55.555241 1160 raft_consensus.cc:2238] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.555374 1160 raft_consensus.cc:2267] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.555320 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_429cde0f50a9446491261ae0938375dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.555778 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_429cde0f50a9446491261ae0938375dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.556234 1160 ts_tablet_manager.cc:1905] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.556253 1166 ts_tablet_manager.cc:1918] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.556335 1166 log.cc:1198] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/72a455033b674e5a94fc37d0275eb313
I20250109 20:52:55.556599 1166 ts_tablet_manager.cc:1939] T 72a455033b674e5a94fc37d0275eb313 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.558132 455 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet af8c0bf27e994e6aa4cbedca16febe12 (table default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]). Expected version 1 got 0
I20250109 20:52:55.559072 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e020eeb03e041ed977779a07de5bb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.528976 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5f945b345d4c029798756e63867156 (DEFAULT_TABLE table=default.Table_3b054fb299c84cf49c242e3ccb44daf8 [id=a8d9617e1d1d4488abe06c7065bfee48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.560024 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5f945b345d4c029798756e63867156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.561517 1160 ts_tablet_manager.cc:1918] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.561573 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0cbc3b8995704eaf99466fd1d5d0e124" } modify_external_catalogs: true
I20250109 20:52:55.561654 1160 log.cc:1198] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ecb55a61d2e94cbb88d2f44fff1aa8d9
I20250109 20:52:55.561749 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0cbc3b8995704eaf99466fd1d5d0e124" } modify_external_catalogs: true
I20250109 20:52:55.562032 1160 ts_tablet_manager.cc:1939] T ecb55a61d2e94cbb88d2f44fff1aa8d9 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:55.562506 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ecb55a61d2e94cbb88d2f44fff1aa8d9 (table default.Table_58060dd58dce4f7a8afa4d8c891c3ee3 [id=9a08f00bc4e74fc5861932c6f7cbbc58]) successfully deleted
I20250109 20:52:55.567492 890 tablet_service.cc:1467] Processing CreateTablet for tablet 440e6af656094411b094c680c561dfc2 (DEFAULT_TABLE table=default.Table_6dc6acbd34db412caf3281183b3c0f74 [id=df33959601234af5b72e96a7580b3381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.567914 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 440e6af656094411b094c680c561dfc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.569432 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 649) took 1480 ms. Trace:
I20250109 20:52:55.569545 1166 rpcz_store.cc:276] 0109 20:52:54.089188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.092237 (+ 3049us) service_pool.cc:225] Handling call
0109 20:52:55.544207 (+1451970us) ts_tablet_manager.cc:1145] Deleting tablet 72a455033b674e5a94fc37d0275eb313
0109 20:52:55.544216 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.547672 (+ 3456us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.556175 (+ 8503us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.569407 (+ 13232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451829}
I20250109 20:52:55.570096 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 663) took 1368 ms. Trace:
I20250109 20:52:55.570168 1166 rpcz_store.cc:276] 0109 20:52:54.201613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.201682 (+ 69us) service_pool.cc:225] Handling call
0109 20:52:55.569984 (+1368302us) ts_tablet_manager.cc:1145] Deleting tablet b9f5b6ca7ef84234b3d667d50ee5b8cd
0109 20:52:55.569991 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.570081 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1368122}
I20250109 20:52:55.570459 1166 tablet_replica.cc:331] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.570631 1166 raft_consensus.cc:2238] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.570772 1166 raft_consensus.cc:2267] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.571199 1166 ts_tablet_manager.cc:1905] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.571700 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 72a455033b674e5a94fc37d0275eb313 (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]) successfully deleted
W20250109 20:52:55.571942 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b9f5b6ca7ef84234b3d667d50ee5b8cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9f5b6ca7ef84234b3d667d50ee5b8cd already in progress: creating tablet
I20250109 20:52:55.572247 1165 tablet_bootstrap.cc:492] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:55.572403 1165 ts_tablet_manager.cc:1397] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250109 20:52:55.573077 1165 raft_consensus.cc:357] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.573302 1165 raft_consensus.cc:383] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.573390 1165 raft_consensus.cc:738] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.573642 1165 consensus_queue.cc:260] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.575102 1165 ts_tablet_manager.cc:1428] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 20:52:55.576143 1165 tablet_bootstrap.cc:492] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:55.576367 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 440e6af656094411b094c680c561dfc2 (DEFAULT_TABLE table=default.Table_6dc6acbd34db412caf3281183b3c0f74 [id=df33959601234af5b72e96a7580b3381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.576689 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 440e6af656094411b094c680c561dfc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.576946 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5a36dfb8096e4cf79407414ba122d0e5" } modify_external_catalogs: true
I20250109 20:52:55.577131 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5a36dfb8096e4cf79407414ba122d0e5" } modify_external_catalogs: true
I20250109 20:52:55.583495 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d" } modify_external_catalogs: true
I20250109 20:52:55.584932 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b1e5a40512794ab9ad99471666d8f7b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.585446 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1e5a40512794ab9ad99471666d8f7b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.585556 1165 tablet_bootstrap.cc:654] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.586879 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc2c7b12f945ea8307a187e1553226" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:55.588039 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 0cede2a1882946a181b18d66f12c5114 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.588119 1176 tablet_bootstrap.cc:492] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.588227 1176 ts_tablet_manager.cc:1397] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20250109 20:52:55.588243 914 raft_consensus.cc:3054] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.589850 752 tablet_service.cc:1467] Processing CreateTablet for tablet f4f6196d820f4aecaff2701aaf56faf5 (DEFAULT_TABLE table=default.Table_77c9fc51089f43c0a1c2db8a4d77573f [id=0f3291f9fb3a4ac583055a2fb53980a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.590030 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8190cb332af34a398ae8970567562d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.592518 854 leader_election.cc:304] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:55.594190 1176 raft_consensus.cc:357] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.594435 1176 raft_consensus.cc:383] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.594492 1176 raft_consensus.cc:738] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.594679 1176 consensus_queue.cc:260] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.594718 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe8a9e5b5fe406b988126cd1859eabc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:55.595232 990 leader_election.cc:343] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.595343 990 leader_election.cc:304] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.595741 1878 raft_consensus.cc:2743] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:55.598410 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f6196d820f4aecaff2701aaf56faf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.599509 754 tablet_service.cc:1467] Processing CreateTablet for tablet 1591374d521a47ebb405a812fe656b19 (DEFAULT_TABLE table=default.Table_814844402c5e4defbb723b986171a2a2 [id=a4138fd6ff5246d582fc0d75b890c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.600143 758 tablet_service.cc:1467] Processing CreateTablet for tablet 6e020eeb03e041ed977779a07de5bb80 (DEFAULT_TABLE table=default.Table_3873238920c845cb8338b7a36ef75274 [id=8bec14b8b4b74d9c87fe86950f5128e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.600565 741 tablet_service.cc:1467] Processing CreateTablet for tablet 55bf7303e7354aa99a7b5bb489f442fe (DEFAULT_TABLE table=default.Table_881894033dec46b3862c1185df595c40 [id=a3ccfed03acd4b2d9fe9bcef1683dc4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.601310 757 tablet_service.cc:1467] Processing CreateTablet for tablet a792f8c86cd041deb4186424efd0e68a (DEFAULT_TABLE table=default.Table_8fdf081b7e1d4d55b7fc92577bfa1b3a [id=3dc32b9bc767455aba6f8dc97d7034ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.601753 1166 ts_tablet_manager.cc:1918] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.601836 1166 log.cc:1198] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4015c0e66dfd47bfb9855f82044f01e7
I20250109 20:52:55.602099 1166 ts_tablet_manager.cc:1939] T 4015c0e66dfd47bfb9855f82044f01e7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.603268 753 tablet_service.cc:1467] Processing CreateTablet for tablet 11dbbc9ceb1a4252b1605560bdb9b62a (DEFAULT_TABLE table=default.Table_532e9f42a7084d3c9037828c563a3f7b [id=b520db9fb09a4f1180e081161f6762ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.603569 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1591374d521a47ebb405a812fe656b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.603613 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0ec1d30b67674facb8b92a2c1a408073
I20250109 20:52:55.606702 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0ee970cc32dd483fa588ff287b7314e5
I20250109 20:52:55.607511 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.607506 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d" } modify_external_catalogs: true
I20250109 20:52:55.608000 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55bf7303e7354aa99a7b5bb489f442fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.608460 1926 tablet.cc:1721] T b31b9785ee21418ba13b426ba94cfa9b P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250109 20:52:55.610217 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e020eeb03e041ed977779a07de5bb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.610143 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 664) took 1404 ms. Trace:
I20250109 20:52:55.610385 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a792f8c86cd041deb4186424efd0e68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.611768 1166 rpcz_store.cc:276] 0109 20:52:54.205340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.205399 (+ 59us) service_pool.cc:225] Handling call
0109 20:52:55.570425 (+1365026us) ts_tablet_manager.cc:1145] Deleting tablet 4015c0e66dfd47bfb9855f82044f01e7
0109 20:52:55.570432 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.573581 (+ 3149us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.601662 (+ 28081us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.610125 (+ 8463us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364856}
I20250109 20:52:55.612251 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 665) took 1403 ms. Trace:
I20250109 20:52:55.612340 1166 rpcz_store.cc:276] 0109 20:52:54.208307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.208370 (+ 63us) service_pool.cc:225] Handling call
0109 20:52:55.612118 (+1403748us) ts_tablet_manager.cc:1145] Deleting tablet 0f68bc0a564c4210bb29172a8759ebf3
0109 20:52:55.612126 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.612212 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403534}
I20250109 20:52:55.612735 1176 ts_tablet_manager.cc:1428] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250109 20:52:55.612802 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 668) took 1365 ms. Trace:
I20250109 20:52:55.613042 1176 tablet_bootstrap.cc:492] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.612903 1166 rpcz_store.cc:276] 0109 20:52:54.247213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.247274 (+ 61us) service_pool.cc:225] Handling call
0109 20:52:55.612651 (+1365377us) ts_tablet_manager.cc:1145] Deleting tablet 89719a0d926c458d825371782980468c
0109 20:52:55.612659 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.612768 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365160}
I20250109 20:52:55.613188 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4015c0e66dfd47bfb9855f82044f01e7 (table default.Table_532d28a5835d421ab3f13d74939ec028 [id=2a15fab6f69a4e399c87563e7e5be289]) successfully deleted
W20250109 20:52:55.613390 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0f68bc0a564c4210bb29172a8759ebf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f68bc0a564c4210bb29172a8759ebf3 already in progress: creating tablet
I20250109 20:52:55.613423 1166 tablet_replica.cc:331] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:55.613575 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 89719a0d926c458d825371782980468c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89719a0d926c458d825371782980468c already in progress: creating tablet
I20250109 20:52:55.613878 1166 raft_consensus.cc:2238] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.613890 745 tablet_service.cc:1467] Processing CreateTablet for tablet 440e6af656094411b094c680c561dfc2 (DEFAULT_TABLE table=default.Table_6dc6acbd34db412caf3281183b3c0f74 [id=df33959601234af5b72e96a7580b3381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.614041 1166 raft_consensus.cc:2267] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.614539 1166 ts_tablet_manager.cc:1905] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.614879 1176 tablet_bootstrap.cc:654] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.615794 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee970cc32dd483fa588ff287b7314e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.617079 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 6787f74100894fe6a7053a6d92ea7b96 with delete_type TABLET_DATA_DELETED (replaced by tablet d8bc15fecd8542e49ff5a0eb22211ec9) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.618425 1925 tablet.cc:1721] T b31b9785ee21418ba13b426ba94cfa9b P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250109 20:52:55.620529 750 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5f945b345d4c029798756e63867156 (DEFAULT_TABLE table=default.Table_3b054fb299c84cf49c242e3ccb44daf8 [id=a8d9617e1d1d4488abe06c7065bfee48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.620963 748 tablet_service.cc:1467] Processing CreateTablet for tablet 062f1d04adc14599ace583896a3117e0 (DEFAULT_TABLE table=default.Table_84dfe41ea9994c1aa62b04e2ec77e4c9 [id=5df146fe156743f0a506e95469ae9964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.621215 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b0450d88e3e4ccaae4e5157ce5c80c1) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.621402 1157 tablet_replica.cc:331] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:55.621589 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11dbbc9ceb1a4252b1605560bdb9b62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.622354 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cf48fa3edeeb4675a6e04b039bdee9ee" } modify_external_catalogs: true
I20250109 20:52:55.622579 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cf48fa3edeeb4675a6e04b039bdee9ee" } modify_external_catalogs: true
I20250109 20:52:55.623701 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4661e42d6cc949cd9e496a9aa3119d6f" } modify_external_catalogs: true
I20250109 20:52:55.623865 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4661e42d6cc949cd9e496a9aa3119d6f" } modify_external_catalogs: true
I20250109 20:52:55.625299 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee970cc32dd483fa588ff287b7314e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.625537 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.626317 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0ee970cc32dd483fa588ff287b7314e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ee970cc32dd483fa588ff287b7314e5 already in progress: creating tablet
I20250109 20:52:55.626483 1157 raft_consensus.cc:2238] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:55.626559 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0ec1d30b67674facb8b92a2c1a408073 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ec1d30b67674facb8b92a2c1a408073 already in progress: creating tablet
I20250109 20:52:55.626657 1157 raft_consensus.cc:2267] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.626852 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 440e6af656094411b094c680c561dfc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.627288 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0cede2a1882946a181b18d66f12c5114
I20250109 20:52:55.627698 453 catalog_manager.cc:6596] ReplaceTablet: tablet 0cede2a1882946a181b18d66f12c5114 deleted and replaced by tablet 1343e8e822684fea8a13d8df9ec33779
I20250109 20:52:55.628578 1176 tablet_bootstrap.cc:492] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.628819 1176 ts_tablet_manager.cc:1397] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20250109 20:52:55.629487 1176 raft_consensus.cc:357] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.629662 1176 raft_consensus.cc:383] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.629729 1176 raft_consensus.cc:738] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.630014 1176 consensus_queue.cc:260] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.630640 1176 ts_tablet_manager.cc:1428] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:55.631033 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cede2a1882946a181b18d66f12c5114 with delete_type TABLET_DATA_DELETED (replaced by tablet 1343e8e822684fea8a13d8df9ec33779) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.631574 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 402280f75e51463a865c0fb521d0fd8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.631736 1157 ts_tablet_manager.cc:1905] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.633795 1160 tablet_replica.cc:331] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:55.633952 1160 raft_consensus.cc:2238] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.634076 1160 raft_consensus.cc:2267] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.634476 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 062f1d04adc14599ace583896a3117e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.634694 1160 ts_tablet_manager.cc:1905] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.635200 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2a83e584f7694f5e9407449f73f9c907
I20250109 20:52:55.635875 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet cc1f638e7cb840368e521a2c7fd05c0d
W20250109 20:52:55.636270 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6787f74100894fe6a7053a6d92ea7b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6787f74100894fe6a7053a6d92ea7b96 already in progress: creating tablet
I20250109 20:52:55.637696 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a83e584f7694f5e9407449f73f9c907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.637786 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5f945b345d4c029798756e63867156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.637827 1166 ts_tablet_manager.cc:1918] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.637939 1166 log.cc:1198] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/a8da9781ab32429297d0c875c1971501
I20250109 20:52:55.637921 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cede2a1882946a181b18d66f12c5114 with delete_type TABLET_DATA_DELETED (replaced by tablet 1343e8e822684fea8a13d8df9ec33779) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.638226 1166 ts_tablet_manager.cc:1939] T a8da9781ab32429297d0c875c1971501 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.639478 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.640921 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a83e584f7694f5e9407449f73f9c907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.647818 1160 ts_tablet_manager.cc:1918] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.648675 1160 log.cc:1198] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/402280f75e51463a865c0fb521d0fd8a
I20250109 20:52:55.649406 1160 ts_tablet_manager.cc:1939] T 402280f75e51463a865c0fb521d0fd8a P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:55.650514 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 402280f75e51463a865c0fb521d0fd8a (table default.Table_e740022397a8451e830a159fa021e461 [id=24e621d6ed7e4a78a53808f4a8c2dd23]) successfully deleted
I20250109 20:52:55.656819 1158 tablet_bootstrap.cc:492] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:55.662519 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0a8d43084a0e45b48d75867cfcbe83e1" } new_table_name: "default.Table_0f59949f33e14c1d8be3879d0043e1b9" modify_external_catalogs: true
I20250109 20:52:55.662542 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 16c9f7274a7542e89a19bd4afcf89205 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:55.665549 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2a83e584f7694f5e9407449f73f9c907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a83e584f7694f5e9407449f73f9c907 already in progress: creating tablet
W20250109 20:52:55.666363 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0cede2a1882946a181b18d66f12c5114 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cede2a1882946a181b18d66f12c5114 already in progress: creating tablet
I20250109 20:52:55.668839 1165 tablet_bootstrap.cc:492] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:55.668970 1165 ts_tablet_manager.cc:1397] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.093s user 0.014s sys 0.000s
I20250109 20:52:55.669097 1176 tablet_bootstrap.cc:492] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.669660 1165 raft_consensus.cc:357] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.669839 1165 raft_consensus.cc:383] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.669889 1165 raft_consensus.cc:738] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.670095 1165 consensus_queue.cc:260] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.670753 1176 tablet_bootstrap.cc:654] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.671983 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 697) took 1197 ms. Trace:
I20250109 20:52:55.672071 1166 rpcz_store.cc:276] 0109 20:52:54.474782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.569460 (+ 94678us) service_pool.cc:225] Handling call
0109 20:52:55.613371 (+1043911us) ts_tablet_manager.cc:1145] Deleting tablet a8da9781ab32429297d0c875c1971501
0109 20:52:55.613380 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.617367 (+ 3987us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.637720 (+ 20353us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.671933 (+ 34213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043721}
I20250109 20:52:55.672369 1166 tablet_replica.cc:331] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.672537 1166 raft_consensus.cc:2238] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.672704 1166 raft_consensus.cc:2267] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.674343 1166 ts_tablet_manager.cc:1905] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.676458 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet a8da9781ab32429297d0c875c1971501 (table default.Table_4c30a9425b844ee187a29bc12cd2704e [id=8fddac7b2a084c15b76824d25240859d]) successfully deleted
I20250109 20:52:55.680094 1166 ts_tablet_manager.cc:1918] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.685427 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d72a28b8d2b94712805f7299f0ccccf6" } new_table_name: "default.Table_4f20d7d11abf4421aa8bc8467e69cc37" modify_external_catalogs: true
I20250109 20:52:55.688302 453 master_service.cc:946] ReplaceTablet: received request to replace tablet c83b61e879fc4979a85c872f2e6ab88c from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.688807 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5b967deae0f94c0cbfb8ff72d3fe8e1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.689334 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b967deae0f94c0cbfb8ff72d3fe8e1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.689492 1166 log.cc:1198] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4d2ad301c11943b69b59be18d6775513
I20250109 20:52:55.689867 1166 ts_tablet_manager.cc:1939] T 4d2ad301c11943b69b59be18d6775513 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.690008 1176 tablet_bootstrap.cc:492] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.690171 1176 ts_tablet_manager.cc:1397] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
W20250109 20:52:55.691107 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8190cb332af34a398ae8970567562d0e consensus state has no leader
I20250109 20:52:55.690960 1176 raft_consensus.cc:357] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.691747 1176 raft_consensus.cc:383] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.691821 1176 raft_consensus.cc:738] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.691824 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_55c51e87298e485d81fb45c134a1c858"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.692010 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 700) took 1216 ms. Trace:
I20250109 20:52:55.692066 1176 consensus_queue.cc:260] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.692111 1166 rpcz_store.cc:276] 0109 20:52:54.475020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.580584 (+105564us) service_pool.cc:225] Handling call
0109 20:52:55.672304 (+1091720us) ts_tablet_manager.cc:1145] Deleting tablet 4d2ad301c11943b69b59be18d6775513
0109 20:52:55.672312 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.677071 (+ 4759us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.679994 (+ 2923us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.691973 (+ 11979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1048930}
W20250109 20:52:55.692235 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55c51e87298e485d81fb45c134a1c858 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.692400 1166 tablet_replica.cc:331] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.691006 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_27dce06a871149eb94e251f6e60e048d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.695003 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27dce06a871149eb94e251f6e60e048d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.695612 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4d2ad301c11943b69b59be18d6775513 (table default.Table_4f8710d177654a9c81c383e7aac0841b [id=3a1cd7e71eab4f90ae054b885cee1626]) successfully deleted
I20250109 20:52:55.696808 1176 ts_tablet_manager.cc:1428] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:52:55.697480 1176 tablet_bootstrap.cc:492] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.697822 1878 raft_consensus.cc:491] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.697966 1878 raft_consensus.cc:513] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.698504 1878 leader_election.cc:290] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:55.698623 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c8d8d771b56407d94fcb23d76796039 consensus state has no leader
I20250109 20:52:55.699266 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cede2a1882946a181b18d66f12c5114 with delete_type TABLET_DATA_DELETED (replaced by tablet 1343e8e822684fea8a13d8df9ec33779) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.699970 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee970cc32dd483fa588ff287b7314e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.700165 1158 ts_tablet_manager.cc:1397] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.232s user 0.000s sys 0.005s
I20250109 20:52:55.701174 1158 raft_consensus.cc:357] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.701434 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a83e584f7694f5e9407449f73f9c907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.701563 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8c0bf27e994e6aa4cbedca16febe12" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:55.701930 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_40c6cee0e5254b01b5329858620643fb" } new_table_name: "default.Table_09eafff3b75242c3822e14ee2d781828" modify_external_catalogs: true
I20250109 20:52:55.702033 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8c0bf27e994e6aa4cbedca16febe12" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:55.702447 990 leader_election.cc:343] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:55.702836 989 leader_election.cc:343] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.702920 989 leader_election.cc:304] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.703536 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_23ea215227ba4c4ea8414587c0e2d085"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.703939 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23ea215227ba4c4ea8414587c0e2d085 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.705260 1157 ts_tablet_manager.cc:1918] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.705363 1157 log.cc:1198] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/7d7539cf9ece4befaae6c521d471ea57
I20250109 20:52:55.705682 1157 ts_tablet_manager.cc:1939] T 7d7539cf9ece4befaae6c521d471ea57 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:55.707042 1166 raft_consensus.cc:2238] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.707180 1166 raft_consensus.cc:2267] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.709829 1166 ts_tablet_manager.cc:1905] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.710037 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8b869f8172ed4fcf815c58fc296752c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.710484 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b869f8172ed4fcf815c58fc296752c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.712409 1158 raft_consensus.cc:383] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.712488 1158 raft_consensus.cc:738] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.712705 1158 consensus_queue.cc:260] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.713424 918 raft_consensus.cc:3054] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.716073 918 raft_consensus.cc:1270] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:55.716775 1678 consensus_queue.cc:1035] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:55.716962 1158 ts_tablet_manager.cc:1428] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
W20250109 20:52:55.721768 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9629fe30752f49478afad9b93dc235eb consensus state has no leader
I20250109 20:52:55.721941 1910 raft_consensus.cc:491] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.722047 1925 tablet.cc:1721] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:55.722071 1910 raft_consensus.cc:513] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.722541 1910 leader_election.cc:290] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.723202 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e6e2b3b9a4f9f9d08361ba1cbb5ba" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.725437 1176 tablet_bootstrap.cc:654] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:55.725647 712 leader_election.cc:343] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.731175 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 7d7539cf9ece4befaae6c521d471ea57 (table default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]) successfully deleted
W20250109 20:52:55.731446 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0ec1d30b67674facb8b92a2c1a408073 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ec1d30b67674facb8b92a2c1a408073 already in progress: creating tablet
W20250109 20:52:55.731694 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0cede2a1882946a181b18d66f12c5114 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cede2a1882946a181b18d66f12c5114 already in progress: creating tablet
W20250109 20:52:55.731911 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0ee970cc32dd483fa588ff287b7314e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ee970cc32dd483fa588ff287b7314e5 already in progress: creating tablet
W20250109 20:52:55.732184 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2a83e584f7694f5e9407449f73f9c907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a83e584f7694f5e9407449f73f9c907 already in progress: creating tablet
I20250109 20:52:55.732663 1158 tablet_bootstrap.cc:492] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:55.734359 1158 tablet_bootstrap.cc:654] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.742272 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8817ccacbbc345248314d8aa7d5cc7ba" } new_table_name: "default.Table_5b8da4f333124137bceb5a3b48b64d41" modify_external_catalogs: true
I20250109 20:52:55.742909 1166 ts_tablet_manager.cc:1918] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.743036 1166 log.cc:1198] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4025cdef634d4626a33f01c362a1388a
I20250109 20:52:55.743331 1166 ts_tablet_manager.cc:1939] T 4025cdef634d4626a33f01c362a1388a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.746776 1951 raft_consensus.cc:2743] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:55.746982 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 44153.5us
I20250109 20:52:55.747443 1051 raft_consensus.cc:1270] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250109 20:52:55.747445 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2769e5906c604bd5b14df79db765ec61 consensus state has no leader
I20250109 20:52:55.747911 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e6e2b3b9a4f9f9d08361ba1cbb5ba" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:55.748067 1051 raft_consensus.cc:2463] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:55.748781 1910 consensus_queue.cc:1035] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:55.750137 713 leader_election.cc:304] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.750573 1910 raft_consensus.cc:2798] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:55.750651 1910 raft_consensus.cc:491] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:55.750712 1910 raft_consensus.cc:3054] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.753465 1910 raft_consensus.cc:513] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.754393 1910 leader_election.cc:290] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.755075 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e6e2b3b9a4f9f9d08361ba1cbb5ba" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:55.755730 712 leader_election.cc:343] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.756661 1158 tablet_bootstrap.cc:492] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:55.756883 1158 ts_tablet_manager.cc:1397] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
W20250109 20:52:55.759617 712 consensus_peers.cc:487] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:55.760324 1158 raft_consensus.cc:357] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.760938 1158 raft_consensus.cc:383] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.761046 1158 raft_consensus.cc:738] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.761287 1158 consensus_queue.cc:260] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.762063 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e6e2b3b9a4f9f9d08361ba1cbb5ba" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:55.762228 1055 raft_consensus.cc:3054] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.763175 1176 tablet_bootstrap.cc:492] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.763278 1176 ts_tablet_manager.cc:1397] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20250109 20:52:55.763898 1176 raft_consensus.cc:357] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.764056 1176 raft_consensus.cc:383] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.764094 1176 raft_consensus.cc:738] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.764582 1055 raft_consensus.cc:2463] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:55.764897 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 698) took 1290 ms. Trace:
I20250109 20:52:55.764981 1166 rpcz_store.cc:276] 0109 20:52:54.474857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.569673 (+ 94816us) service_pool.cc:225] Handling call
0109 20:52:55.692341 (+1122668us) ts_tablet_manager.cc:1145] Deleting tablet 4025cdef634d4626a33f01c362a1388a
0109 20:52:55.692349 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.724510 (+ 32161us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.742789 (+ 18279us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.764878 (+ 22089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061263}
I20250109 20:52:55.765399 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4025cdef634d4626a33f01c362a1388a (table default.Table_d52320f2397e4526ae456263f71bcda2 [id=0b96e2cb7e234f498b8ef5e9a09e9a67]) successfully deleted
I20250109 20:52:55.765880 1158 ts_tablet_manager.cc:1428] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250109 20:52:55.766431 713 leader_election.cc:304] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.766642 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ed7be8358b1c4e17acb5190ca02be11b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.767163 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed7be8358b1c4e17acb5190ca02be11b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.767716 1950 tablet.cc:1721] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:55.767892 1166 tablet_replica.cc:331] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.768046 1166 raft_consensus.cc:2238] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.768167 1166 raft_consensus.cc:2267] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.768263 1823 raft_consensus.cc:2743] T e6cc2c7b12f945ea8307a187e1553226 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:55.768671 1166 ts_tablet_manager.cc:1905] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.769230 1910 raft_consensus.cc:2798] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:55.770080 1910 raft_consensus.cc:695] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:55.770148 1949 raft_consensus.cc:491] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09833ea46b7141d58c3899dace69e49f)
I20250109 20:52:55.770231 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1a922ef3992c4ec088cfdf98b527dcf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.770308 1949 raft_consensus.cc:513] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:55.770370 1910 consensus_queue.cc:237] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:55.770704 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a922ef3992c4ec088cfdf98b527dcf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.770960 1949 leader_election.cc:290] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:55.772212 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e615ce8e124cfe9432778b6b58e117" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:55.772652 1158 tablet_bootstrap.cc:492] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:55.773049 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet e6cc2c7b12f945ea8307a187e1553226 (table default.Table_19a1e320401641a383cf7c3fc20b4713 [id=e0737f359bfa4154b4fb71617efc829f]). Expected version 2 got 0
I20250109 20:52:55.774050 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_429cde0f50a9446491261ae0938375dc" } new_table_name: "default.Table_6dde16e3825445a38efe43db92927674" modify_external_catalogs: true
I20250109 20:52:55.774128 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e615ce8e124cfe9432778b6b58e117" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:55.774863 1158 tablet_bootstrap.cc:654] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.775033 854 leader_election.cc:304] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:55.775620 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c58a40d4195d443d82138e5eedbb7b93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.776072 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c58a40d4195d443d82138e5eedbb7b93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.777756 1949 raft_consensus.cc:2743] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:52:55.779161 1176 consensus_queue.cc:260] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.782055 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 06e615ce8e124cfe9432778b6b58e117 (table default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]). Expected version 1 got 0
I20250109 20:52:55.782235 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:55.784615 1176 ts_tablet_manager.cc:1428] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
W20250109 20:52:55.786652 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7204d58f2cf640059ad827e11f62912a has no consensus state
I20250109 20:52:55.787946 1158 tablet_bootstrap.cc:492] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:55.788579 1158 ts_tablet_manager.cc:1397] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250109 20:52:55.789336 1158 raft_consensus.cc:357] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.789520 1176 tablet_bootstrap.cc:492] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.791159 1158 raft_consensus.cc:383] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.792289 1158 raft_consensus.cc:738] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.792560 1158 consensus_queue.cc:260] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.792874 1957 tablet.cc:1721] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:55.796461 1158 ts_tablet_manager.cc:1428] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250109 20:52:55.796890 1176 tablet_bootstrap.cc:654] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.799785 1926 tablet.cc:1721] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:55.800405 1158 tablet_bootstrap.cc:492] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:55.809332 1158 tablet_bootstrap.cc:654] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.813385 1166 ts_tablet_manager.cc:1918] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.813482 1166 log.cc:1198] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/a58d439254f8418091e3f23c8c274240
I20250109 20:52:55.813733 1166 ts_tablet_manager.cc:1939] T a58d439254f8418091e3f23c8c274240 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.814409 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 699) took 1339 ms. Trace:
I20250109 20:52:55.814497 1166 rpcz_store.cc:276] 0109 20:52:54.474935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.572748 (+ 97813us) service_pool.cc:225] Handling call
0109 20:52:55.767851 (+1195103us) ts_tablet_manager.cc:1145] Deleting tablet a58d439254f8418091e3f23c8c274240
0109 20:52:55.767860 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.794870 (+ 27010us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.813311 (+ 18441us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.814375 (+ 1064us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136057}
I20250109 20:52:55.814893 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet a58d439254f8418091e3f23c8c274240 (table default.Table_4c9330ec3e904978bb8b1f0869a0c532 [id=fbf8fa58eaee4147b61be651b770935b]) successfully deleted
I20250109 20:52:55.819160 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 84f3c6c8764a4af1a9111e147c9c6ac3 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.819437 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c86baf1a40a34eb4beeff2b71b73afde" } modify_external_catalogs: true
I20250109 20:52:55.819628 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c86baf1a40a34eb4beeff2b71b73afde" } modify_external_catalogs: true
I20250109 20:52:55.819511 1176 tablet_bootstrap.cc:492] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:55.820122 1176 ts_tablet_manager.cc:1397] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250109 20:52:55.820554 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_486f2f72d50e49cdb7e8a1c022cb439d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.821040 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_486f2f72d50e49cdb7e8a1c022cb439d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.821027 1176 raft_consensus.cc:357] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.821182 1176 raft_consensus.cc:383] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.821223 1176 raft_consensus.cc:738] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.821410 1176 consensus_queue.cc:260] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.821875 1176 ts_tablet_manager.cc:1428] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:52:55.822047 1878 raft_consensus.cc:491] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.822134 1878 raft_consensus.cc:513] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.823155 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b405d16cc62444f2ad94fb69bb9fb6ee" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:55.823801 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_94a783531318417989d76ed0dd1d8bdc" } modify_external_catalogs: true
I20250109 20:52:55.823988 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_94a783531318417989d76ed0dd1d8bdc" } modify_external_catalogs: true
I20250109 20:52:55.825399 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b405d16cc62444f2ad94fb69bb9fb6ee" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.826018 1158 tablet_bootstrap.cc:492] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
W20250109 20:52:55.823666 990 leader_election.cc:343] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.826135 1158 ts_tablet_manager.cc:1397] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
W20250109 20:52:55.826124 989 leader_election.cc:343] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.826210 989 leader_election.cc:304] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.823055 1878 leader_election.cc:290] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:55.826406 1165 ts_tablet_manager.cc:1428] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.157s user 0.013s sys 0.015s
I20250109 20:52:55.826416 1951 raft_consensus.cc:2743] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:55.824602 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 701) took 1014 ms. Trace:
I20250109 20:52:55.826884 1158 raft_consensus.cc:357] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.827100 1158 raft_consensus.cc:383] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.827188 1158 raft_consensus.cc:738] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.826614 1166 rpcz_store.cc:276] 0109 20:52:54.810380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.810418 (+ 38us) service_pool.cc:225] Handling call
0109 20:52:55.824490 (+1014072us) ts_tablet_manager.cc:1145] Deleting tablet ffd3a1b1d3e44feda6dbaa8556e377fc
0109 20:52:55.824500 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.824587 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1013866}
W20250109 20:52:55.827708 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ffd3a1b1d3e44feda6dbaa8556e377fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffd3a1b1d3e44feda6dbaa8556e377fc already in progress: creating tablet
I20250109 20:52:55.827419 1158 consensus_queue.cc:260] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.829512 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 702) took 1018 ms. Trace:
I20250109 20:52:55.829588 1166 rpcz_store.cc:276] 0109 20:52:54.810633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.810660 (+ 27us) service_pool.cc:225] Handling call
0109 20:52:55.827669 (+1017009us) ts_tablet_manager.cc:1145] Deleting tablet a6630c808e354ecdb168332d5c3ea8da
0109 20:52:55.829412 (+ 1743us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.829497 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1016879}
W20250109 20:52:55.831421 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a6630c808e354ecdb168332d5c3ea8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6630c808e354ecdb168332d5c3ea8da already in progress: creating tablet
I20250109 20:52:55.832031 1158 ts_tablet_manager.cc:1428] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:52:55.833432 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_7e98b984f52d4023ac3365b3e23f26e2 [id=4d18944cc25a4da6a21e3040370599e0] alter complete (version 5)
I20250109 20:52:55.834429 1910 raft_consensus.cc:491] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.834542 1910 raft_consensus.cc:513] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.832747 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 703) took 1021 ms. Trace:
I20250109 20:52:55.835100 1910 leader_election.cc:290] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.835063 1166 rpcz_store.cc:276] 0109 20:52:54.810776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.810815 (+ 39us) service_pool.cc:225] Handling call
0109 20:52:55.831692 (+1020877us) ts_tablet_manager.cc:1145] Deleting tablet 6461f153a5994412ba05c3b687963fd1
0109 20:52:55.831699 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.831772 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020474}
W20250109 20:52:55.835439 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6461f153a5994412ba05c3b687963fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6461f153a5994412ba05c3b687963fd1 already in progress: creating tablet
I20250109 20:52:55.835739 1158 tablet_bootstrap.cc:492] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:55.836846 1166 tablet_replica.cc:331] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:55.836968 1158 tablet_bootstrap.cc:654] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.837000 1166 raft_consensus.cc:2238] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.837134 1166 raft_consensus.cc:2267] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.837575 1166 ts_tablet_manager.cc:1905] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.840492 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881ea20c17904d669c8a1fbd766ba559" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:55.841418 712 leader_election.cc:343] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.842221 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881ea20c17904d669c8a1fbd766ba559" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:55.842413 1051 raft_consensus.cc:2463] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:55.842682 713 leader_election.cc:304] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.843122 1910 raft_consensus.cc:2798] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:55.843179 1910 raft_consensus.cc:491] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:55.843225 1910 raft_consensus.cc:3054] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.843251 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_472ddec2dea04b1d83cb147944a488c9" } modify_external_catalogs: true
I20250109 20:52:55.843454 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_472ddec2dea04b1d83cb147944a488c9" } modify_external_catalogs: true
I20250109 20:52:55.843990 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f48177e873b84ff3bc7fabaa2ac6f672"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.844314 1158 tablet_bootstrap.cc:492] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:55.844439 1158 ts_tablet_manager.cc:1397] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
W20250109 20:52:55.844481 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f48177e873b84ff3bc7fabaa2ac6f672 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.845232 1158 raft_consensus.cc:357] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.845388 1158 raft_consensus.cc:383] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.845439 1158 raft_consensus.cc:738] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:55.845526 1910 raft_consensus.cc:513] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.845665 1158 consensus_queue.cc:260] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.846704 1910 leader_election.cc:290] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.848199 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881ea20c17904d669c8a1fbd766ba559" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:55.848696 712 leader_election.cc:343] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.848989 1158 ts_tablet_manager.cc:1428] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:55.849769 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_3585fc2fc449422880a7ebe625c22d09 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 1 got 0
I20250109 20:52:55.849967 451 catalog_manager.cc:5526] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:55.850347 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:55.851608 1165 tablet_bootstrap.cc:492] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:55.852317 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881ea20c17904d669c8a1fbd766ba559" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:55.852447 1051 raft_consensus.cc:3054] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.853991 1158 tablet_bootstrap.cc:492] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:55.856390 1158 tablet_bootstrap.cc:654] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.857221 1051 raft_consensus.cc:2463] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:55.858767 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 84f3c6c8764a4af1a9111e147c9c6ac3
I20250109 20:52:55.859025 713 leader_election.cc:304] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.859228 454 catalog_manager.cc:6596] ReplaceTablet: tablet 84f3c6c8764a4af1a9111e147c9c6ac3 deleted and replaced by tablet af6d061fd1e34117bd8ba609c0638d48
I20250109 20:52:55.859412 1910 raft_consensus.cc:2798] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:55.859529 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 with delete_type TABLET_DATA_DELETED (replaced by tablet af6d061fd1e34117bd8ba609c0638d48) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.859867 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet beca5f1ed9d3434fb7de90cd74a6f09b
I20250109 20:52:55.859984 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 with delete_type TABLET_DATA_DELETED (replaced by tablet af6d061fd1e34117bd8ba609c0638d48) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.860473 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0b0450d88e3e4ccaae4e5157ce5c80c1
I20250109 20:52:55.860687 753 tablet_service.cc:1514] Processing DeleteTablet for tablet beca5f1ed9d3434fb7de90cd74a6f09b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.861056 1910 raft_consensus.cc:695] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
W20250109 20:52:55.861194 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f3c6c8764a4af1a9111e147c9c6ac3 already in progress: creating tablet
I20250109 20:52:55.861243 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet beca5f1ed9d3434fb7de90cd74a6f09b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.861330 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.861469 1910 consensus_queue.cc:237] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.861693 1165 tablet_bootstrap.cc:654] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.861742 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.861846 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 with delete_type TABLET_DATA_DELETED (replaced by tablet af6d061fd1e34117bd8ba609c0638d48) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.862013 890 tablet_service.cc:1514] Processing DeleteTablet for tablet beca5f1ed9d3434fb7de90cd74a6f09b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.862196 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet beca5f1ed9d3434fb7de90cd74a6f09b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet beca5f1ed9d3434fb7de90cd74a6f09b already in progress: creating tablet
W20250109 20:52:55.862407 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 already in progress: creating tablet
I20250109 20:52:55.862443 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.862572 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f3c6c8764a4af1a9111e147c9c6ac3 already in progress: creating tablet
W20250109 20:52:55.862789 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet beca5f1ed9d3434fb7de90cd74a6f09b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet beca5f1ed9d3434fb7de90cd74a6f09b already in progress: creating tablet
I20250109 20:52:55.862882 1960 raft_consensus.cc:491] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.863072 1960 raft_consensus.cc:513] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.863356 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3e2f412cd17540698bf3788a7fc9cca8" } modify_external_catalogs: true
I20250109 20:52:55.863536 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3e2f412cd17540698bf3788a7fc9cca8" } modify_external_catalogs: true
I20250109 20:52:55.863934 1960 leader_election.cc:290] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:55.864049 1960 raft_consensus.cc:491] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.864044 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6619b3dce3ed4325a4871c3d40b42c9d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:55.864131 1960 raft_consensus.cc:513] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:55.864466 990 leader_election.cc:343] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:55.863534 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 already in progress: creating tablet
I20250109 20:52:55.864689 1960 leader_election.cc:290] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:55.865008 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1630e4438e4337ba026b41f630a480" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:55.865437 990 leader_election.cc:343] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.865984 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6619b3dce3ed4325a4871c3d40b42c9d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.868227 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c83b61e879fc4979a85c872f2e6ab88c
I20250109 20:52:55.868391 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1630e4438e4337ba026b41f630a480" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.868683 453 catalog_manager.cc:6596] ReplaceTablet: tablet c83b61e879fc4979a85c872f2e6ab88c deleted and replaced by tablet c3d9f992800e4a41a7af7e2751214eaa
W20250109 20:52:55.868835 989 leader_election.cc:343] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.868922 989 leader_election.cc:304] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.869067 753 tablet_service.cc:1514] Processing DeleteTablet for tablet c83b61e879fc4979a85c872f2e6ab88c with delete_type TABLET_DATA_DELETED (replaced by tablet c3d9f992800e4a41a7af7e2751214eaa) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.869109 1951 raft_consensus.cc:491] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.869160 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c83b61e879fc4979a85c872f2e6ab88c with delete_type TABLET_DATA_DELETED (replaced by tablet c3d9f992800e4a41a7af7e2751214eaa) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.869215 1951 raft_consensus.cc:513] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:55.869626 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c83b61e879fc4979a85c872f2e6ab88c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c83b61e879fc4979a85c872f2e6ab88c already in progress: creating tablet
I20250109 20:52:55.869670 1951 leader_election.cc:290] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:55.869779 1951 raft_consensus.cc:2743] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:55.869846 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c83b61e879fc4979a85c872f2e6ab88c with delete_type TABLET_DATA_DELETED (replaced by tablet c3d9f992800e4a41a7af7e2751214eaa) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.870158 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a7c04f3f948e1a5e486d5ccd48ccc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:55.870349 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c83b61e879fc4979a85c872f2e6ab88c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c83b61e879fc4979a85c872f2e6ab88c already in progress: creating tablet
W20250109 20:52:55.870602 990 leader_election.cc:343] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:55.870869 989 leader_election.cc:343] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.870959 989 leader_election.cc:304] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.871511 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705a7c04f3f948e1a5e486d5ccd48ccc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:55.872699 989 leader_election.cc:343] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.872822 989 leader_election.cc:304] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.874048 1951 raft_consensus.cc:2743] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:55.874202 1951 raft_consensus.cc:2743] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:55.874285 448 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:55.874698 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38] alter complete (version 1)
I20250109 20:52:55.877023 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 92235a34fb1541fda31a05314346d27b
I20250109 20:52:55.876271 1176 tablet_bootstrap.cc:492] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:55.877877 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 92235a34fb1541fda31a05314346d27b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.878041 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 92235a34fb1541fda31a05314346d27b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:55.878407 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 92235a34fb1541fda31a05314346d27b because the tablet was not found. No further retry: Not found: Tablet not found: 92235a34fb1541fda31a05314346d27b
I20250109 20:52:55.878525 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 92235a34fb1541fda31a05314346d27b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.878584 1176 tablet_bootstrap.cc:654] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:55.879067 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 92235a34fb1541fda31a05314346d27b because the tablet was not found. No further retry: Not found: Tablet not found: 92235a34fb1541fda31a05314346d27b
I20250109 20:52:55.879871 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 with delete_type TABLET_DATA_DELETED (replaced by tablet e16a1d9a5b374b8dbf725eb32ffadb9e) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.880307 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dbccae0af24a629600a92e7eadfee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.880308 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc with delete_type TABLET_DATA_DELETED (replaced by tablet cd23012af5574f37944f9e26c07d05d2) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.880837 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.881024 1157 tablet_replica.cc:331] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:55.881204 1157 raft_consensus.cc:2238] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.881325 1157 raft_consensus.cc:2267] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.881831 1157 ts_tablet_manager.cc:1905] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.882352 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b221e048f6f6463e84e630386c304a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.886353 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:52:55.886663 451 catalog_manager.cc:5526] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:55.887132 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:55.888286 1157 ts_tablet_manager.cc:1918] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.888435 1157 log.cc:1198] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4015c0e66dfd47bfb9855f82044f01e7
I20250109 20:52:55.888818 1157 ts_tablet_manager.cc:1939] T 4015c0e66dfd47bfb9855f82044f01e7 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:55.890041 655 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 92235a34fb1541fda31a05314346d27b
I20250109 20:52:55.892221 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 92235a34fb1541fda31a05314346d27b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.892923 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 92235a34fb1541fda31a05314346d27b because the tablet was not found. No further retry: Not found: Tablet not found: 92235a34fb1541fda31a05314346d27b
I20250109 20:52:55.894155 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4015c0e66dfd47bfb9855f82044f01e7 (table default.Table_532d28a5835d421ab3f13d74939ec028 [id=2a15fab6f69a4e399c87563e7e5be289]) successfully deleted
I20250109 20:52:55.894866 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_35995197ab7d4324ae39cee214b8316f" } new_table_name: "default.Table_993baf73c83a4aa2afdae6a2afa40f0d" modify_external_catalogs: true
I20250109 20:52:55.895999 1157 tablet_replica.cc:331] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:55.896348 1157 raft_consensus.cc:2238] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.896536 1157 raft_consensus.cc:2267] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.896934 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet face19003d0042669e3f22553f9fe19a
I20250109 20:52:55.897540 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 92235a34fb1541fda31a05314346d27b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.897686 750 tablet_service.cc:1514] Processing DeleteTablet for tablet face19003d0042669e3f22553f9fe19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.898198 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet face19003d0042669e3f22553f9fe19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.898576 1157 ts_tablet_manager.cc:1905] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.898874 897 tablet_service.cc:1514] Processing DeleteTablet for tablet face19003d0042669e3f22553f9fe19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.899609 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet face19003d0042669e3f22553f9fe19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet face19003d0042669e3f22553f9fe19a already in progress: creating tablet
I20250109 20:52:55.899847 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet faeec3dbcaff4de98e94357f20e7e8ad
I20250109 20:52:55.900640 750 tablet_service.cc:1514] Processing DeleteTablet for tablet faeec3dbcaff4de98e94357f20e7e8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:55.900655 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet faeec3dbcaff4de98e94357f20e7e8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.900990 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0be3c25688ad443cadd4f1c2c068f2c4" } new_table_name: "default.Table_9281c8ca884748bfb0e463cff8bee9e7" modify_external_catalogs: true
I20250109 20:52:55.902784 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 92235a34fb1541fda31a05314346d27b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:55.903177 1023 tablet_service.cc:1467] Processing CreateTablet for tablet ac867ce748f749d88885ef13654983ec (DEFAULT_TABLE table=default.Table_aff5b59bbc5445d8b655981c6f4ca537 [id=779dca0771d84adfbc5632801ddd8ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.903148 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 24c5c15866bb4c39863309f6c5b075d2 (DEFAULT_TABLE table=default.Table_83f4a695b41741509248430f3d916085 [id=99e5b569eeaf402c87be2097bc8031e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.903357 753 tablet_service.cc:1467] Processing CreateTablet for tablet 97d3eca4eaf54ab4a3ac588a128934a9 (DEFAULT_TABLE table=default.Table_5b5c1f8eb7b14686abcfdab2e28fa57d [id=f315f40d62854fb7bef554be46bef8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.903491 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac867ce748f749d88885ef13654983ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.903489 745 tablet_service.cc:1467] Processing CreateTablet for tablet bd642e5ec83a4da88eaa2eea3ffca362 (DEFAULT_TABLE table=default.Table_cc87fbc188284d3f9099de392fb15757 [id=f9364fb6fc124cb192a7f1968a02bf6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.903560 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 97d3eca4eaf54ab4a3ac588a128934a9 (DEFAULT_TABLE table=default.Table_5b5c1f8eb7b14686abcfdab2e28fa57d [id=f315f40d62854fb7bef554be46bef8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.903653 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d3eca4eaf54ab4a3ac588a128934a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.903836 750 tablet_service.cc:1467] Processing CreateTablet for tablet 24c5c15866bb4c39863309f6c5b075d2 (DEFAULT_TABLE table=default.Table_83f4a695b41741509248430f3d916085 [id=99e5b569eeaf402c87be2097bc8031e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.904101 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c5c15866bb4c39863309f6c5b075d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.904981 755 tablet_service.cc:1467] Processing CreateTablet for tablet ac867ce748f749d88885ef13654983ec (DEFAULT_TABLE table=default.Table_aff5b59bbc5445d8b655981c6f4ca537 [id=779dca0771d84adfbc5632801ddd8ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.905254 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac867ce748f749d88885ef13654983ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.908308 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd642e5ec83a4da88eaa2eea3ffca362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.908936 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d3eca4eaf54ab4a3ac588a128934a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.913058 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c5c15866bb4c39863309f6c5b075d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.916235 880 tablet_service.cc:1467] Processing CreateTablet for tablet 97d3eca4eaf54ab4a3ac588a128934a9 (DEFAULT_TABLE table=default.Table_5b5c1f8eb7b14686abcfdab2e28fa57d [id=f315f40d62854fb7bef554be46bef8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.916419 893 tablet_service.cc:1467] Processing CreateTablet for tablet bd642e5ec83a4da88eaa2eea3ffca362 (DEFAULT_TABLE table=default.Table_cc87fbc188284d3f9099de392fb15757 [id=f9364fb6fc124cb192a7f1968a02bf6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.916603 887 tablet_service.cc:1467] Processing CreateTablet for tablet ac867ce748f749d88885ef13654983ec (DEFAULT_TABLE table=default.Table_aff5b59bbc5445d8b655981c6f4ca537 [id=779dca0771d84adfbc5632801ddd8ba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.916960 897 tablet_service.cc:1514] Processing DeleteTablet for tablet faeec3dbcaff4de98e94357f20e7e8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.916419 890 tablet_service.cc:1467] Processing CreateTablet for tablet 24c5c15866bb4c39863309f6c5b075d2 (DEFAULT_TABLE table=default.Table_83f4a695b41741509248430f3d916085 [id=99e5b569eeaf402c87be2097bc8031e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.918396 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a4fac4fff616461d9e0b116e98bdd5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.918879 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4fac4fff616461d9e0b116e98bdd5e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:55.919461 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet faeec3dbcaff4de98e94357f20e7e8ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faeec3dbcaff4de98e94357f20e7e8ad already in progress: creating tablet
I20250109 20:52:55.919924 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8e9896750c594aa78160152def6a4f15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.920375 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e9896750c594aa78160152def6a4f15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.921787 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2d8d4a28c8d342e7be15b81a4a9785ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.922181 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d8d4a28c8d342e7be15b81a4a9785ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.923592 1157 ts_tablet_manager.cc:1918] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:55.924052 1157 log.cc:1198] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/b221e048f6f6463e84e630386c304a05
I20250109 20:52:55.924692 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8a7392fff9c0401fb5c701ee92cf9c41" } new_table_name: "default.Table_7336cda1ce164722949bdefb9f26e7cd" modify_external_catalogs: true
I20250109 20:52:55.925500 1157 ts_tablet_manager.cc:1939] T b221e048f6f6463e84e630386c304a05 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:55.926097 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_735a572f26234eb29120c917e17347c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.926470 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_735a572f26234eb29120c917e17347c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.927611 1165 tablet_bootstrap.cc:492] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:55.927845 1165 ts_tablet_manager.cc:1397] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250109 20:52:55.928254 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_df479c8972204ae1a07040b5e2dea99c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.928617 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet b221e048f6f6463e84e630386c304a05 (table default.Table_43ed51a629174e3bbaad39c5d8f43ef7 [id=ce77658acdf34b7e90401b0982c045fe]) successfully deleted
W20250109 20:52:55.928682 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df479c8972204ae1a07040b5e2dea99c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.929471 1165 raft_consensus.cc:357] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
W20250109 20:52:55.929733 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 92235a34fb1541fda31a05314346d27b because the tablet was not found. No further retry: Not found: Tablet not found: 92235a34fb1541fda31a05314346d27b
I20250109 20:52:55.929919 1165 raft_consensus.cc:383] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:55.930130 1165 raft_consensus.cc:738] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:52:55.931181 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet face19003d0042669e3f22553f9fe19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet face19003d0042669e3f22553f9fe19a already in progress: creating tablet
I20250109 20:52:55.931612 1165 consensus_queue.cc:260] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
W20250109 20:52:55.932171 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet faeec3dbcaff4de98e94357f20e7e8ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faeec3dbcaff4de98e94357f20e7e8ad already in progress: creating tablet
W20250109 20:52:55.933650 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64da106b64354bd3b5c6cacef7677631 consensus state has no leader
I20250109 20:52:55.938948 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:55.939594 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 46d4cad7594346dfafdee8e4c49d74e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46d4cad7594346dfafdee8e4c49d74e6 already in progress: creating tablet
I20250109 20:52:55.940889 1165 ts_tablet_manager.cc:1428] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250109 20:52:55.945420 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d4cad7594346dfafdee8e4c49d74e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:55.945657 1160 tablet_replica.cc:331] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:55.945828 1160 raft_consensus.cc:2238] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:55.945989 1160 raft_consensus.cc:2267] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:55.946997 1023 tablet_service.cc:1467] Processing CreateTablet for tablet bd642e5ec83a4da88eaa2eea3ffca362 (DEFAULT_TABLE table=default.Table_cc87fbc188284d3f9099de392fb15757 [id=f9364fb6fc124cb192a7f1968a02bf6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.947424 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd642e5ec83a4da88eaa2eea3ffca362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.949231 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cb4c99bfeae9477485df9f52b78ea759" } new_table_name: "default.Table_385822fe7219471bacdd1ad2dbe7d891" modify_external_catalogs: true
I20250109 20:52:55.949455 1165 tablet_bootstrap.cc:492] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:55.957302 1926 tablet.cc:1721] T ff4f256d59a2459e95be0f950a5a86eb P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:55.958943 897 tablet_service.cc:1467] Processing CreateTablet for tablet a3cf6dbe0a0b472aafd2cfdbc308e6b6 (DEFAULT_TABLE table=default.Table_2df8048c9d2f4c6e85562681c0ed0b94 [id=818a6cb3dbb04b9fb82bb1a5d446a41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.959452 1160 ts_tablet_manager.cc:1905] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:55.960359 1678 raft_consensus.cc:491] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.960479 1678 raft_consensus.cc:513] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.961024 1678 leader_election.cc:290] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.961730 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be44ffc9e371452f8d5f4174fbfa752a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.962453 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be44ffc9e371452f8d5f4174fbfa752a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:55.962606 1055 raft_consensus.cc:2463] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:55.963335 745 tablet_service.cc:1467] Processing CreateTablet for tablet a3cf6dbe0a0b472aafd2cfdbc308e6b6 (DEFAULT_TABLE table=default.Table_2df8048c9d2f4c6e85562681c0ed0b94 [id=818a6cb3dbb04b9fb82bb1a5d446a41a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:55.963402 712 leader_election.cc:343] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.963814 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cf6dbe0a0b472aafd2cfdbc308e6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.964967 1051 raft_consensus.cc:1270] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:52:55.968335 1951 raft_consensus.cc:491] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:55.968438 1951 raft_consensus.cc:513] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.968969 1951 leader_election.cc:290] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:55.969049 712 consensus_peers.cc:487] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:55.969491 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e4bfa188b545349f358adfd0bc714b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:55.969604 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d3eca4eaf54ab4a3ac588a128934a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.969892 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd642e5ec83a4da88eaa2eea3ffca362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.970230 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cf6dbe0a0b472aafd2cfdbc308e6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.972802 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c5c15866bb4c39863309f6c5b075d2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:55.973198 990 leader_election.cc:343] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.973750 713 leader_election.cc:304] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.974220 1910 raft_consensus.cc:2798] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:55.974285 1910 raft_consensus.cc:491] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:55.974342 1910 raft_consensus.cc:3054] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.975796 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 (DEFAULT_TABLE table=default.Table_676e3bec2c46486b82b8e636ba06a4d3 [id=7104da780d49424794f146b3feee3673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.976099 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ddb127ac2ad49b9820a91a0f6b010a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.976801 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac867ce748f749d88885ef13654983ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.977336 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e4bfa188b545349f358adfd0bc714b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:55.977725 1678 consensus_queue.cc:1035] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:52:55.978406 989 leader_election.cc:343] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.978494 989 leader_election.cc:304] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:55.976603 1910 raft_consensus.cc:513] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.979621 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be44ffc9e371452f8d5f4174fbfa752a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:55.980028 712 leader_election.cc:343] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:55.980741 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ebdd59dd13424ecf93c997fa5919477d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:55.981237 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebdd59dd13424ecf93c997fa5919477d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.982961 1951 raft_consensus.cc:2743] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:55.985289 1910 leader_election.cc:290] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:55.985651 893 tablet_service.cc:1467] Processing CreateTablet for tablet e02af8aad8cb495e8ebeabe9c46c8c52 (DEFAULT_TABLE table=default.Table_7b61b7cfa1b14d94a0150b0590511425 [id=cddaca980c7946c2b378a335289b004d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:55.985883 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be44ffc9e371452f8d5f4174fbfa752a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:55.986024 1055 raft_consensus.cc:3054] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:55.989372 1925 tablet.cc:1721] T ff4f256d59a2459e95be0f950a5a86eb P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:55.989583 1055 raft_consensus.cc:2463] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:55.989845 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3873238920c845cb8338b7a36ef75274" } new_table_name: "default.Table_1f4279f07e8b4e0a86928bdb9a67bbf6" modify_external_catalogs: true
I20250109 20:52:55.989964 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_17e950d4133f4d2a8eeab7f7af0b10c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.990005 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_12b56043fe0e4f4681a9522f8d62d94c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:55.986333 1160 ts_tablet_manager.cc:1918] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:55.990478 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17e950d4133f4d2a8eeab7f7af0b10c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.990499 1160 log.cc:1198] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/46d4cad7594346dfafdee8e4c49d74e6
I20250109 20:52:55.990772 1160 ts_tablet_manager.cc:1939] T 46d4cad7594346dfafdee8e4c49d74e6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:55.991245 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 46d4cad7594346dfafdee8e4c49d74e6 (table default.Table_49b7725a1f7647cd9b136c55bd8ed301 [id=ccd17bbcb44f49a3b2eae73c6ace6c4c]) successfully deleted
I20250109 20:52:55.991524 713 leader_election.cc:304] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:55.991631 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12b56043fe0e4f4681a9522f8d62d94c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:55.991955 1910 raft_consensus.cc:2798] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:55.992094 1910 raft_consensus.cc:695] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:55.992283 1910 consensus_queue.cc:237] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:55.992817 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e02af8aad8cb495e8ebeabe9c46c8c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.987690 1165 tablet_bootstrap.cc:654] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:55.997967 1166 ts_tablet_manager.cc:1918] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:55.998122 1166 log.cc:1198] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/d6b06e8616df461d89c58d414fc0cfbd
I20250109 20:52:55.998476 1166 ts_tablet_manager.cc:1939] T d6b06e8616df461d89c58d414fc0cfbd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:55.999955 750 tablet_service.cc:1467] Processing CreateTablet for tablet e02af8aad8cb495e8ebeabe9c46c8c52 (DEFAULT_TABLE table=default.Table_7b61b7cfa1b14d94a0150b0590511425 [id=cddaca980c7946c2b378a335289b004d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.000378 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e02af8aad8cb495e8ebeabe9c46c8c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:55.988082 897 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 (DEFAULT_TABLE table=default.Table_676e3bec2c46486b82b8e636ba06a4d3 [id=7104da780d49424794f146b3feee3673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.002980 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ddb127ac2ad49b9820a91a0f6b010a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.003265 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 575f3cb9f61143ab949fa85f47edcafb from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:55.988616 1957 tablet.cc:1721] T ff4f256d59a2459e95be0f950a5a86eb P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:56.008798 1023 tablet_service.cc:1467] Processing CreateTablet for tablet a3cf6dbe0a0b472aafd2cfdbc308e6b6 (DEFAULT_TABLE table=default.Table_2df8048c9d2f4c6e85562681c0ed0b94 [id=818a6cb3dbb04b9fb82bb1a5d446a41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.009218 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cf6dbe0a0b472aafd2cfdbc308e6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.015712 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 8de735008fec4af0a7181cc10426842c (DEFAULT_TABLE table=default.Table_85420187c03540a3832414efe59f28d6 [id=a7887ab067874d7ba1050b0112a0f80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.015815 1678 consensus_queue.cc:1035] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:56.016007 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de735008fec4af0a7181cc10426842c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.016623 1176 tablet_bootstrap.cc:492] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.016767 1176 ts_tablet_manager.cc:1397] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.141s user 0.012s sys 0.012s
I20250109 20:52:56.017400 1176 raft_consensus.cc:357] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.017555 1176 raft_consensus.cc:383] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.017602 1176 raft_consensus.cc:738] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.019387 897 tablet_service.cc:1467] Processing CreateTablet for tablet 8de735008fec4af0a7181cc10426842c (DEFAULT_TABLE table=default.Table_85420187c03540a3832414efe59f28d6 [id=a7887ab067874d7ba1050b0112a0f80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.019667 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de735008fec4af0a7181cc10426842c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.024406 1016 tablet_service.cc:1467] Processing CreateTablet for tablet e02af8aad8cb495e8ebeabe9c46c8c52 (DEFAULT_TABLE table=default.Table_7b61b7cfa1b14d94a0150b0590511425 [id=cddaca980c7946c2b378a335289b004d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.025781 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e02af8aad8cb495e8ebeabe9c46c8c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.028728 750 tablet_service.cc:1467] Processing CreateTablet for tablet 8de735008fec4af0a7181cc10426842c (DEFAULT_TABLE table=default.Table_85420187c03540a3832414efe59f28d6 [id=a7887ab067874d7ba1050b0112a0f80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.029356 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de735008fec4af0a7181cc10426842c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.030810 745 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 (DEFAULT_TABLE table=default.Table_676e3bec2c46486b82b8e636ba06a4d3 [id=7104da780d49424794f146b3feee3673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.032208 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ddb127ac2ad49b9820a91a0f6b010a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.037874 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 57e56f30e6f3450a8426cd40db89e009 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:56.043524 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c34129b9dd664780b2e0c613259f6357 (DEFAULT_TABLE table=default.Table_655023e44afe42d597ce3e6cd3a1b9bf [id=ff9ae2d477da4b6793ec4f84dcfba58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.043900 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34129b9dd664780b2e0c613259f6357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.045943 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_993baf73c83a4aa2afdae6a2afa40f0d [id=c6b6eaf39b88459490f459e9268f4980] alter complete (version 1)
I20250109 20:52:56.048696 890 tablet_service.cc:1467] Processing CreateTablet for tablet c708cc308b42444e9edbe7198e78a49f (DEFAULT_TABLE table=default.Table_54a6148851094ffc9fb3923fb869a9e1 [id=f9b7997f5dc5457ba7abdf3109dc18e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.049104 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c708cc308b42444e9edbe7198e78a49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.050323 1017 tablet_service.cc:1467] Processing CreateTablet for tablet c708cc308b42444e9edbe7198e78a49f (DEFAULT_TABLE table=default.Table_54a6148851094ffc9fb3923fb869a9e1 [id=f9b7997f5dc5457ba7abdf3109dc18e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.050662 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c708cc308b42444e9edbe7198e78a49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.054838 897 tablet_service.cc:1467] Processing CreateTablet for tablet 51b760fcb4fc466ba49d456892d43f3c (DEFAULT_TABLE table=default.Table_dfc57ba40e274837bd12f0b1085b2eed [id=ec5195f1191e4e36a9401713493fca07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.055207 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b760fcb4fc466ba49d456892d43f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.055848 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a82e1240e6aa4513b7008ddeb3dff288"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.056326 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a82e1240e6aa4513b7008ddeb3dff288 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.057159 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 704) took 1220 ms. Trace:
I20250109 20:52:56.057353 1166 rpcz_store.cc:276] 0109 20:52:54.836688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.836807 (+ 119us) service_pool.cc:225] Handling call
0109 20:52:55.836810 (+1000003us) ts_tablet_manager.cc:1145] Deleting tablet d6b06e8616df461d89c58d414fc0cfbd
0109 20:52:55.836818 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:55.959292 (+122474us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:55.997878 (+ 38586us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.057105 (+ 59227us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57395,"tablet-delete.queue_time_us":999828}
I20250109 20:52:56.057955 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet d6b06e8616df461d89c58d414fc0cfbd (table default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]) successfully deleted
I20250109 20:52:56.058494 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 705) took 1138 ms. Trace:
I20250109 20:52:56.058719 1166 rpcz_store.cc:276] 0109 20:52:54.920139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.920200 (+ 61us) service_pool.cc:225] Handling call
0109 20:52:56.058279 (+1138079us) ts_tablet_manager.cc:1145] Deleting tablet bdce283727d1491ba0968a17c1c00d87
0109 20:52:56.058288 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.058446 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136827}
W20250109 20:52:56.059424 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet bdce283727d1491ba0968a17c1c00d87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdce283727d1491ba0968a17c1c00d87 already in progress: creating tablet
I20250109 20:52:56.059942 1166 tablet_replica.cc:331] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.060295 1166 raft_consensus.cc:2238] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.060604 1166 raft_consensus.cc:2267] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.060895 1960 raft_consensus.cc:491] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.061000 1960 raft_consensus.cc:513] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.061518 1960 leader_election.cc:290] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:56.061949 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91dbccae0af24a629600a92e7eadfee7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:56.062481 989 leader_election.cc:343] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:56.063877 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e020eeb03e041ed977779a07de5bb80 consensus state has no leader
I20250109 20:52:56.068853 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cd252806276c4bc28ada21e4c72c6c08" } modify_external_catalogs: true
I20250109 20:52:56.069085 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cd252806276c4bc28ada21e4c72c6c08" } modify_external_catalogs: true
I20250109 20:52:56.070648 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d6cafa4ebffe4a048881a673e64692d6" } modify_external_catalogs: true
I20250109 20:52:56.070828 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d6cafa4ebffe4a048881a673e64692d6" } modify_external_catalogs: true
I20250109 20:52:56.076655 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6f7d71e007e84fafa7d6634d3ee5e122"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:56.076869 753 tablet_service.cc:1467] Processing CreateTablet for tablet c34129b9dd664780b2e0c613259f6357 (DEFAULT_TABLE table=default.Table_655023e44afe42d597ce3e6cd3a1b9bf [id=ff9ae2d477da4b6793ec4f84dcfba58e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:56.077095 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f7d71e007e84fafa7d6634d3ee5e122 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.077183 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34129b9dd664780b2e0c613259f6357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.078467 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a34821f5c1e44d4cb26f0daa699282d0" } modify_external_catalogs: true
I20250109 20:52:56.080521 451 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 2608f48f5950466e9ab2d12ee87503cb (table default.Table_7336cda1ce164722949bdefb9f26e7cd [id=17d12fac83d6463fb91fab6729a08139]). Expected version 2 got 0
I20250109 20:52:56.080698 451 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:56.080739 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a34821f5c1e44d4cb26f0daa699282d0" } modify_external_catalogs: true
I20250109 20:52:56.092986 1176 consensus_queue.cc:260] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.093658 1176 ts_tablet_manager.cc:1428] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.077s user 0.001s sys 0.000s
I20250109 20:52:56.093921 1176 tablet_bootstrap.cc:492] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.095386 1176 tablet_bootstrap.cc:654] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.101128 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 575f3cb9f61143ab949fa85f47edcafb
I20250109 20:52:56.101723 455 catalog_manager.cc:6596] ReplaceTablet: tablet 575f3cb9f61143ab949fa85f47edcafb deleted and replaced by tablet 4f1499e571354faf9952af5288b5c3fa
I20250109 20:52:56.101943 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 575f3cb9f61143ab949fa85f47edcafb with delete_type TABLET_DATA_DELETED (replaced by tablet 4f1499e571354faf9952af5288b5c3fa) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.102110 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 57e56f30e6f3450a8426cd40db89e009
I20250109 20:52:56.102531 454 catalog_manager.cc:6596] ReplaceTablet: tablet 57e56f30e6f3450a8426cd40db89e009 deleted and replaced by tablet eaea4f239cce41a997496a1c94810ac9
W20250109 20:52:56.102730 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 575f3cb9f61143ab949fa85f47edcafb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575f3cb9f61143ab949fa85f47edcafb already in progress: creating tablet
I20250109 20:52:56.103228 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e56f30e6f3450a8426cd40db89e009 with delete_type TABLET_DATA_DELETED (replaced by tablet eaea4f239cce41a997496a1c94810ac9) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.104153 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 575f3cb9f61143ab949fa85f47edcafb with delete_type TABLET_DATA_DELETED (replaced by tablet 4f1499e571354faf9952af5288b5c3fa) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.104153 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e56f30e6f3450a8426cd40db89e009 with delete_type TABLET_DATA_DELETED (replaced by tablet eaea4f239cce41a997496a1c94810ac9) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:56.104524 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 57e56f30e6f3450a8426cd40db89e009 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57e56f30e6f3450a8426cd40db89e009 already in progress: creating tablet
I20250109 20:52:56.101333 893 tablet_service.cc:1467] Processing CreateTablet for tablet c34129b9dd664780b2e0c613259f6357 (DEFAULT_TABLE table=default.Table_655023e44afe42d597ce3e6cd3a1b9bf [id=ff9ae2d477da4b6793ec4f84dcfba58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.109254 1960 raft_consensus.cc:491] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.109397 1960 raft_consensus.cc:513] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.109953 1960 leader_election.cc:290] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:56.110852 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 575f3cb9f61143ab949fa85f47edcafb with delete_type TABLET_DATA_DELETED (replaced by tablet 4f1499e571354faf9952af5288b5c3fa) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.112025 755 tablet_service.cc:1467] Processing CreateTablet for tablet c708cc308b42444e9edbe7198e78a49f (DEFAULT_TABLE table=default.Table_54a6148851094ffc9fb3923fb869a9e1 [id=f9b7997f5dc5457ba7abdf3109dc18e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.112195 1176 tablet_bootstrap.cc:492] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.112331 1176 ts_tablet_manager.cc:1397] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250109 20:52:56.112450 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c708cc308b42444e9edbe7198e78a49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.112875 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91dbccae0af24a629600a92e7eadfee7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:56.113147 1176 raft_consensus.cc:357] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.113360 1176 raft_consensus.cc:383] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.113412 1176 raft_consensus.cc:738] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:52:56.113718 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 575f3cb9f61143ab949fa85f47edcafb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575f3cb9f61143ab949fa85f47edcafb already in progress: creating tablet
I20250109 20:52:56.114257 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e56f30e6f3450a8426cd40db89e009 with delete_type TABLET_DATA_DELETED (replaced by tablet eaea4f239cce41a997496a1c94810ac9) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:56.114969 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 57e56f30e6f3450a8426cd40db89e009 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57e56f30e6f3450a8426cd40db89e009 already in progress: creating tablet
W20250109 20:52:56.115038 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 86853.6us
I20250109 20:52:56.116416 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cc7c3bfc538c44f5a8f958176b1311c7" } modify_external_catalogs: true
I20250109 20:52:56.116624 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cc7c3bfc538c44f5a8f958176b1311c7" } modify_external_catalogs: true
I20250109 20:52:56.117949 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dcb1259a70b64057b0090a78234d47a6" } modify_external_catalogs: true
I20250109 20:52:56.118095 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dcb1259a70b64057b0090a78234d47a6" } modify_external_catalogs: true
I20250109 20:52:56.122658 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 47726b58d6714cd1b46848a16a34ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.124243 751 tablet_service.cc:1467] Processing CreateTablet for tablet 51b760fcb4fc466ba49d456892d43f3c (DEFAULT_TABLE table=default.Table_dfc57ba40e274837bd12f0b1085b2eed [id=ec5195f1191e4e36a9401713493fca07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.124653 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b760fcb4fc466ba49d456892d43f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.130363 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6b7f0230e20d45acaa48894fa41c1219
I20250109 20:52:56.131528 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b7f0230e20d45acaa48894fa41c1219 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.133052 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b7f0230e20d45acaa48894fa41c1219 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:56.133666 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6b7f0230e20d45acaa48894fa41c1219 because the tablet was not found. No further retry: Not found: Tablet not found: 6b7f0230e20d45acaa48894fa41c1219
I20250109 20:52:56.133977 1176 consensus_queue.cc:260] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.137595 1176 ts_tablet_manager.cc:1428] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20250109 20:52:56.139389 1176 tablet_bootstrap.cc:492] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.110327 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47726b58d6714cd1b46848a16a34ff37" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.131100 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b7f0230e20d45acaa48894fa41c1219 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.141877 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6b7f0230e20d45acaa48894fa41c1219 because the tablet was not found. No further retry: Not found: Tablet not found: 6b7f0230e20d45acaa48894fa41c1219
W20250109 20:52:56.142272 989 leader_election.cc:343] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.107443 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34129b9dd664780b2e0c613259f6357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.149060 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d9ef5e30a4624d08b31b37a6c230d437"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.149552 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9ef5e30a4624d08b31b37a6c230d437 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.150914 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b527bb4ba85142d1aabf4d172f9d8e36" } new_table_name: "default.Table_7592f0da4beb49cda4e5e9c6392a085b" modify_external_catalogs: true
I20250109 20:52:56.151026 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_78e828c9c8034bd88102f0763df9f88d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.151485 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78e828c9c8034bd88102f0763df9f88d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.154764 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 71ac8d15d48c49289712ce27768dc27b with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.155433 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b8d4d4aa418a4831a8f6d0fd432b412c
I20250109 20:52:56.156065 897 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d4d4aa418a4831a8f6d0fd432b412c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.156427 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d4d4aa418a4831a8f6d0fd432b412c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:56.156621 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b8d4d4aa418a4831a8f6d0fd432b412c because the tablet was not found. No further retry: Not found: Tablet not found: b8d4d4aa418a4831a8f6d0fd432b412c
W20250109 20:52:56.157536 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b8d4d4aa418a4831a8f6d0fd432b412c because the tablet was not found. No further retry: Not found: Tablet not found: b8d4d4aa418a4831a8f6d0fd432b412c
I20250109 20:52:56.158111 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d4d4aa418a4831a8f6d0fd432b412c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.157832 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3349fa7d981e4c0cbaea5fce12ad7c67" } new_table_name: "default.Table_e6c36b2fd8a749f296aeca3e9191d0f8" modify_external_catalogs: true
I20250109 20:52:56.158145 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8a8d16516a0548d9a55647e97d9e03d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.160008 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a8d16516a0548d9a55647e97d9e03d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:56.164855 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 133107us
I20250109 20:52:56.165568 1158 tablet_bootstrap.cc:492] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:56.165969 1158 ts_tablet_manager.cc:1397] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.312s user 0.035s sys 0.052s
I20250109 20:52:56.167129 1925 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:56.167891 1166 ts_tablet_manager.cc:1905] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:56.171051 990 leader_election.cc:343] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.171164 990 leader_election.cc:304] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.171584 1960 raft_consensus.cc:2743] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.172082 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47726b58d6714cd1b46848a16a34ff37" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:56.172453 990 leader_election.cc:343] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.172559 990 leader_election.cc:304] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.172592 897 tablet_service.cc:1467] Processing CreateTablet for tablet e676bba315ef469da47ad841322f48e9 (DEFAULT_TABLE table=default.Table_d53cb6e3c5514b2e89f54a956be59a83 [id=c1e1116d91ad4ed5954560b6a3bd0122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.172976 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e676bba315ef469da47ad841322f48e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.172921 1957 tablet.cc:1721] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:56.173163 1926 tablet.cc:1721] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:56.173293 1960 raft_consensus.cc:2743] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.173442 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 51b760fcb4fc466ba49d456892d43f3c (DEFAULT_TABLE table=default.Table_dfc57ba40e274837bd12f0b1085b2eed [id=ec5195f1191e4e36a9401713493fca07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.173715 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b760fcb4fc466ba49d456892d43f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.175788 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d664d9d1e75b4d728aadaa5ef1103542" } new_table_name: "default.Table_b16086bc965e4d5c83a477ef56f70482" modify_external_catalogs: true
I20250109 20:52:56.180752 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 49ba8f186a444c8a9cf01d3eb2036c49
I20250109 20:52:56.181318 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5ebf92383a5d419ba3e24ea4eea7ee72
I20250109 20:52:56.181690 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ba8f186a444c8a9cf01d3eb2036c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.182207 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ba8f186a444c8a9cf01d3eb2036c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.184747 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 49ba8f186a444c8a9cf01d3eb2036c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49ba8f186a444c8a9cf01d3eb2036c49 already in progress: creating tablet
I20250109 20:52:56.185828 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8220072e9f42ef90eda39e2382d2df (DEFAULT_TABLE table=default.Table_54e2dbe4b2ac420ba5f1ebd060213e74 [id=bb2caf6627ae41e99654e4bed78ff477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.185904 887 tablet_service.cc:1467] Processing CreateTablet for tablet 5b107b4c26a94898b744654195bfe567 (DEFAULT_TABLE table=default.Table_a997c69220da46df9489620287286652 [id=ffea19003ebc447d964b2ac3ea3a4829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.186115 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8220072e9f42ef90eda39e2382d2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.186057 1158 raft_consensus.cc:357] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.186218 1158 raft_consensus.cc:383] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.186266 1158 raft_consensus.cc:738] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.186509 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3af68b009a4789bc1cdbafc8d44dcf (DEFAULT_TABLE table=default.Table_3ebf8106cfdb42bdbf56ec609f15305d [id=aecb0ed016634a8d86ac527d15fa9f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.186487 1158 consensus_queue.cc:260] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.186863 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b107b4c26a94898b744654195bfe567. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.186920 1158 ts_tablet_manager.cc:1428] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20250109 20:52:56.187134 1158 tablet_bootstrap.cc:492] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:56.188532 1158 tablet_bootstrap.cc:654] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.191854 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3af68b009a4789bc1cdbafc8d44dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.197134 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.197496 1176 tablet_bootstrap.cc:654] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.197667 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.198274 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e676bba315ef469da47ad841322f48e9 (DEFAULT_TABLE table=default.Table_d53cb6e3c5514b2e89f54a956be59a83 [id=c1e1116d91ad4ed5954560b6a3bd0122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.198644 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e676bba315ef469da47ad841322f48e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.202775 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8220072e9f42ef90eda39e2382d2df (DEFAULT_TABLE table=default.Table_54e2dbe4b2ac420ba5f1ebd060213e74 [id=bb2caf6627ae41e99654e4bed78ff477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.203154 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8220072e9f42ef90eda39e2382d2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.203462 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d4c0f80ec2094da0893fb0a92223b1cd
I20250109 20:52:56.204784 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3af68b009a4789bc1cdbafc8d44dcf (DEFAULT_TABLE table=default.Table_3ebf8106cfdb42bdbf56ec609f15305d [id=aecb0ed016634a8d86ac527d15fa9f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.205694 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3af68b009a4789bc1cdbafc8d44dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.206210 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c0f80ec2094da0893fb0a92223b1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.206671 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f03d4e679a1c4793bd83cd979acced88 consensus state has no leader
I20250109 20:52:56.206996 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_be351430ca734ccd895e178828cb82b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.207572 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be351430ca734ccd895e178828cb82b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.209321 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_46c18c9d597b4fb1972f05a7c835e2a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.209744 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46c18c9d597b4fb1972f05a7c835e2a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:56.210068 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d4c0f80ec2094da0893fb0a92223b1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c0f80ec2094da0893fb0a92223b1cd already in progress: creating tablet
I20250109 20:52:56.210336 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.210651 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 49ba8f186a444c8a9cf01d3eb2036c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49ba8f186a444c8a9cf01d3eb2036c49 already in progress: creating tablet
W20250109 20:52:56.211290 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ebf92383a5d419ba3e24ea4eea7ee72 already in progress: creating tablet
I20250109 20:52:56.211475 751 tablet_service.cc:1467] Processing CreateTablet for tablet e676bba315ef469da47ad841322f48e9 (DEFAULT_TABLE table=default.Table_d53cb6e3c5514b2e89f54a956be59a83 [id=c1e1116d91ad4ed5954560b6a3bd0122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.212286 880 tablet_service.cc:1467] Processing CreateTablet for tablet f2d468467478415ba582be6d4a9d989b (DEFAULT_TABLE table=default.Table_d80f0ae2bf3e419faf22aa448e220dfd [id=9507bf0377624fe589232af1333fc3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.212659 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d468467478415ba582be6d4a9d989b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.198129 750 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3af68b009a4789bc1cdbafc8d44dcf (DEFAULT_TABLE table=default.Table_3ebf8106cfdb42bdbf56ec609f15305d [id=aecb0ed016634a8d86ac527d15fa9f0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.213348 887 tablet_service.cc:1467] Processing CreateTablet for tablet 92235a34fb1541fda31a05314346d27b (DEFAULT_TABLE table=default.Table_3e2f412cd17540698bf3788a7fc9cca8 [id=52c5fcbb584142ce9cb91800617172a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.213549 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8220072e9f42ef90eda39e2382d2df (DEFAULT_TABLE table=default.Table_54e2dbe4b2ac420ba5f1ebd060213e74 [id=bb2caf6627ae41e99654e4bed78ff477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.213811 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92235a34fb1541fda31a05314346d27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.214195 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e676bba315ef469da47ad841322f48e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:56.214468 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ebf92383a5d419ba3e24ea4eea7ee72 already in progress: creating tablet
I20250109 20:52:56.214671 897 tablet_service.cc:1467] Processing CreateTablet for tablet b8d4d4aa418a4831a8f6d0fd432b412c (DEFAULT_TABLE table=default.Table_d6cafa4ebffe4a048881a673e64692d6 [id=88447adc99b9463e9a098b9afca7e3b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.214958 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3af68b009a4789bc1cdbafc8d44dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.215934 1166 ts_tablet_manager.cc:1918] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.216030 1166 log.cc:1198] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/773f2c6e6d7f415ab1218f2365c880a2
I20250109 20:52:56.216277 1166 ts_tablet_manager.cc:1939] T 773f2c6e6d7f415ab1218f2365c880a2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.217113 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d4d4aa418a4831a8f6d0fd432b412c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.217646 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c0f80ec2094da0893fb0a92223b1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.217986 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 5b107b4c26a94898b744654195bfe567 (DEFAULT_TABLE table=default.Table_a997c69220da46df9489620287286652 [id=ffea19003ebc447d964b2ac3ea3a4829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.218254 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b107b4c26a94898b744654195bfe567. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.220960 1032 tablet_service.cc:1467] Processing CreateTablet for tablet b8d4d4aa418a4831a8f6d0fd432b412c (DEFAULT_TABLE table=default.Table_d6cafa4ebffe4a048881a673e64692d6 [id=88447adc99b9463e9a098b9afca7e3b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.221252 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d4d4aa418a4831a8f6d0fd432b412c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.221575 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f2d468467478415ba582be6d4a9d989b (DEFAULT_TABLE table=default.Table_d80f0ae2bf3e419faf22aa448e220dfd [id=9507bf0377624fe589232af1333fc3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.221843 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d468467478415ba582be6d4a9d989b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.224217 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ba8f186a444c8a9cf01d3eb2036c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.224268 755 tablet_service.cc:1467] Processing CreateTablet for tablet 5b107b4c26a94898b744654195bfe567 (DEFAULT_TABLE table=default.Table_a997c69220da46df9489620287286652 [id=ffea19003ebc447d964b2ac3ea3a4829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.224567 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 92235a34fb1541fda31a05314346d27b (DEFAULT_TABLE table=default.Table_3e2f412cd17540698bf3788a7fc9cca8 [id=52c5fcbb584142ce9cb91800617172a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.224900 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92235a34fb1541fda31a05314346d27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.227523 1176 tablet_bootstrap.cc:492] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.227658 1176 ts_tablet_manager.cc:1397] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.002s
I20250109 20:52:56.228370 1176 raft_consensus.cc:357] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.228538 1176 raft_consensus.cc:383] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.228592 1176 raft_consensus.cc:738] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.228852 1176 consensus_queue.cc:260] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.230167 754 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c0f80ec2094da0893fb0a92223b1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.230291 758 tablet_service.cc:1467] Processing CreateTablet for tablet f2d468467478415ba582be6d4a9d989b (DEFAULT_TABLE table=default.Table_d80f0ae2bf3e419faf22aa448e220dfd [id=9507bf0377624fe589232af1333fc3df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.230758 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8220072e9f42ef90eda39e2382d2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.230914 741 tablet_service.cc:1467] Processing CreateTablet for tablet b8d4d4aa418a4831a8f6d0fd432b412c (DEFAULT_TABLE table=default.Table_d6cafa4ebffe4a048881a673e64692d6 [id=88447adc99b9463e9a098b9afca7e3b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.231205 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b107b4c26a94898b744654195bfe567. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:56.231268 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 65795.9us
I20250109 20:52:56.231371 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d4d4aa418a4831a8f6d0fd432b412c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.231539 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d468467478415ba582be6d4a9d989b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.234055 890 tablet_service.cc:1467] Processing CreateTablet for tablet b8e935376b8f4bf88f0dee0a739ca096 (DEFAULT_TABLE table=default.Table_a51ccde2af1e4323b06854cff3e6216c [id=80d91c976cc3444fbcac5d39a1d435db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.234378 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e935376b8f4bf88f0dee0a739ca096. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.235842 448 catalog_manager.cc:5526] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:56.236256 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_3585fc2fc449422880a7ebe625c22d09 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 1 got 0
I20250109 20:52:56.237022 896 tablet_service.cc:1467] Processing CreateTablet for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda (DEFAULT_TABLE table=default.Table_89576f5c757a4cc788608386bc1f949b [id=59fad6151b0045a9881254feb4867a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.237361 893 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad5f7eaea8c46b6ba0f83753112094e (DEFAULT_TABLE table=default.Table_0a56e4ac9a02430886c64ffa9268293f [id=7886de19522845a183a79ee660e3422a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.237438 889 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0fdf7cada746ea91e8fb9debcc9475 (DEFAULT_TABLE table=default.Table_0915efe95c4141fa8a554a6b8fb28fea [id=8248bd58bbfd40818738ff7cb4decb60]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:56.238829 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42d3318891db42e68836b9fd3b507de4 consensus state has no leader
I20250109 20:52:56.238837 894 tablet_service.cc:1467] Processing CreateTablet for tablet 1c422468931d4ec18c7d8cf9c392d3a6 (DEFAULT_TABLE table=default.Table_3a97d64313b6410788dafa912c301267 [id=1977e16013a542e087d46f0bc6abf83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.239104 885 tablet_service.cc:1467] Processing CreateTablet for tablet 2495dc49929f45dbb55d2e285d0264df (DEFAULT_TABLE table=default.Table_a2537a166da14ae39e657d27dc555f71 [id=ddaa5387fa2a45f1ac5650dea9e30724]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:56.239246 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3681efa8785480ca65a51b4b40448f1 consensus state has no leader
I20250109 20:52:56.239706 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_385822fe7219471bacdd1ad2dbe7d891 [id=0430588a51ed49ae86290eaab54f79dc] alter complete (version 2)
I20250109 20:52:56.241256 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0fdf7cada746ea91e8fb9debcc9475. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.241429 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c422468931d4ec18c7d8cf9c392d3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.241593 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2495dc49929f45dbb55d2e285d0264df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.241770 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.246922 880 tablet_service.cc:1467] Processing CreateTablet for tablet 64da106b64354bd3b5c6cacef7677631 (DEFAULT_TABLE table=default.Table_9281c8ca884748bfb0e463cff8bee9e7 [id=be6f26583f3a42db87b13b3db06bcda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.247187 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64da106b64354bd3b5c6cacef7677631. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.248163 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad5f7eaea8c46b6ba0f83753112094e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.252723 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c4ae127254154c808ee79dd8e20f184a" } new_table_name: "default.Table_e68def0adf0d46b49a7e236ad95f03e9" modify_external_catalogs: true
I20250109 20:52:56.256501 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b8e935376b8f4bf88f0dee0a739ca096 (DEFAULT_TABLE table=default.Table_a51ccde2af1e4323b06854cff3e6216c [id=80d91c976cc3444fbcac5d39a1d435db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.256866 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e935376b8f4bf88f0dee0a739ca096. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.257011 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda (DEFAULT_TABLE table=default.Table_89576f5c757a4cc788608386bc1f949b [id=59fad6151b0045a9881254feb4867a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.257267 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.259194 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad5f7eaea8c46b6ba0f83753112094e (DEFAULT_TABLE table=default.Table_0a56e4ac9a02430886c64ffa9268293f [id=7886de19522845a183a79ee660e3422a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.259480 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad5f7eaea8c46b6ba0f83753112094e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.260948 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0fdf7cada746ea91e8fb9debcc9475 (DEFAULT_TABLE table=default.Table_0915efe95c4141fa8a554a6b8fb28fea [id=8248bd58bbfd40818738ff7cb4decb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.260967 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 1c422468931d4ec18c7d8cf9c392d3a6 (DEFAULT_TABLE table=default.Table_3a97d64313b6410788dafa912c301267 [id=1977e16013a542e087d46f0bc6abf83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.261235 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0fdf7cada746ea91e8fb9debcc9475. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.261410 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c422468931d4ec18c7d8cf9c392d3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.261637 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 64da106b64354bd3b5c6cacef7677631 (DEFAULT_TABLE table=default.Table_9281c8ca884748bfb0e463cff8bee9e7 [id=be6f26583f3a42db87b13b3db06bcda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.261885 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64da106b64354bd3b5c6cacef7677631. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.264848 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.269665 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b2efe8b52c934acaac976ac5af84531e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.270308 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2efe8b52c934acaac976ac5af84531e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.271646 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 3464ea8be70f494183c1395c9b2b0784 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:56.272856 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5b8da4f333124137bceb5a3b48b64d41" } new_table_name: "default.Table_2a846b026306494b971c14609d9e438e" modify_external_catalogs: true
I20250109 20:52:56.276177 757 tablet_service.cc:1467] Processing CreateTablet for tablet b8e935376b8f4bf88f0dee0a739ca096 (DEFAULT_TABLE table=default.Table_a51ccde2af1e4323b06854cff3e6216c [id=80d91c976cc3444fbcac5d39a1d435db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.276553 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e935376b8f4bf88f0dee0a739ca096. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.276490 742 tablet_service.cc:1467] Processing CreateTablet for tablet 64da106b64354bd3b5c6cacef7677631 (DEFAULT_TABLE table=default.Table_9281c8ca884748bfb0e463cff8bee9e7 [id=be6f26583f3a42db87b13b3db06bcda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.276862 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64da106b64354bd3b5c6cacef7677631. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.276921 754 tablet_service.cc:1467] Processing CreateTablet for tablet 92235a34fb1541fda31a05314346d27b (DEFAULT_TABLE table=default.Table_3e2f412cd17540698bf3788a7fc9cca8 [id=52c5fcbb584142ce9cb91800617172a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.279155 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5cef087cc349eba40c171b326a1f8f (DEFAULT_TABLE table=default.Table_33b6f13c94484781be39523332e10aed [id=9ff0258807514dd790a2cdc5b9bf9be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.279474 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5cef087cc349eba40c171b326a1f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.283747 889 tablet_service.cc:1467] Processing CreateTablet for tablet ef0f27e5506444fd996d9ded42dfacb7 (DEFAULT_TABLE table=default.Table_15d6655499224a0487b0c1f7feb8e3b9 [id=1c0abe6c795949f0a630b57d06edafd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.284374 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0f27e5506444fd996d9ded42dfacb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.285801 894 tablet_service.cc:1467] Processing CreateTablet for tablet b6b23623677b4e088c14af925b4e07d4 (DEFAULT_TABLE table=default.Table_70499a7567fa4d6498705309cde3b257 [id=3ae741f5fe4343188b6f0d10b2a09b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.286080 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b23623677b4e088c14af925b4e07d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.286329 891 tablet_service.cc:1467] Processing CreateTablet for tablet d8bc15fecd8542e49ff5a0eb22211ec9 (DEFAULT_TABLE table=default.Table_d4091d8d942a404cbea7396af4558981 [id=d792d447bf93470bb29ae0b9423783ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.286597 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bc15fecd8542e49ff5a0eb22211ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.288277 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3464ea8be70f494183c1395c9b2b0784
I20250109 20:52:56.288667 453 catalog_manager.cc:6596] ReplaceTablet: tablet 3464ea8be70f494183c1395c9b2b0784 deleted and replaced by tablet e6dba2d7cf654d449a1c9ff85ac1556f
W20250109 20:52:56.288967 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09e412e3a03f42c083032fdf3854ef39 consensus state has no leader
I20250109 20:52:56.290023 892 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f5947674a4d6ea08cc52e72f905f0 (DEFAULT_TABLE table=default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.290232 886 tablet_service.cc:1467] Processing CreateTablet for tablet ece9f0afb8c24a1f95409d21d052a57b (DEFAULT_TABLE table=default.Table_534986b560ef4cbdaf8e21e045a76d29 [id=91e6e240adc342f1a8a20b0cfb817c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.290309 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7f5947674a4d6ea08cc52e72f905f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.290520 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece9f0afb8c24a1f95409d21d052a57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.290709 899 tablet_service.cc:1467] Processing CreateTablet for tablet 530f8e7169f14c84ac2b29314dcc9182 (DEFAULT_TABLE table=default.Table_94c3bc135f0e4e028b93205980b55227 [id=0ef308c37e3647c5917d496cf3151f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.291016 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530f8e7169f14c84ac2b29314dcc9182. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.292923 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3464ea8be70f494183c1395c9b2b0784 with delete_type TABLET_DATA_DELETED (replaced by tablet e6dba2d7cf654d449a1c9ff85ac1556f) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.293638 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 64da106b64354bd3b5c6cacef7677631 (table default.Table_9281c8ca884748bfb0e463cff8bee9e7 [id=be6f26583f3a42db87b13b3db06bcda2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 64da106b64354bd3b5c6cacef7677631 consensus state has no leader
W20250109 20:52:56.293809 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3464ea8be70f494183c1395c9b2b0784 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3464ea8be70f494183c1395c9b2b0784 already in progress: creating tablet
I20250109 20:52:56.295007 1960 raft_consensus.cc:491] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.295140 1960 raft_consensus.cc:513] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.295758 1960 leader_election.cc:290] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:56.296495 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2769e5906c604bd5b14df79db765ec61 consensus state has no leader
I20250109 20:52:56.300738 896 tablet_service.cc:1467] Processing CreateTablet for tablet aec7aec43b744231aa834a14e9d48986 (DEFAULT_TABLE table=default.Table_9fbed746bea74b4086ef2f2dc12674b2 [id=6819b1e50fa34779abd7eb3af5bb49cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.300895 885 tablet_service.cc:1467] Processing CreateTablet for tablet cbbdc5198b314511abfa78d8c8727e18 (DEFAULT_TABLE table=default.Table_e6a87107149f421e93c2ef7e26400ef1 [id=7878e437152b4d90b49864428cd94ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.301033 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b083e6be42f41ecbf23ee57ec17b0a5" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.301095 1176 ts_tablet_manager.cc:1428] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.073s user 0.000s sys 0.002s
I20250109 20:52:56.301402 1176 tablet_bootstrap.cc:492] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:56.301584 989 leader_election.cc:343] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.302908 1176 tablet_bootstrap.cc:654] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.307852 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad5f7eaea8c46b6ba0f83753112094e (DEFAULT_TABLE table=default.Table_0a56e4ac9a02430886c64ffa9268293f [id=7886de19522845a183a79ee660e3422a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.308205 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ad5f7eaea8c46b6ba0f83753112094e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.308887 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92235a34fb1541fda31a05314346d27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.312904 752 tablet_service.cc:1467] Processing CreateTablet for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda (DEFAULT_TABLE table=default.Table_89576f5c757a4cc788608386bc1f949b [id=59fad6151b0045a9881254feb4867a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.313205 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.313956 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 706) took 1393 ms. Trace:
I20250109 20:52:56.314028 1166 rpcz_store.cc:276] 0109 20:52:54.920224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.920354 (+ 130us) service_pool.cc:225] Handling call
0109 20:52:56.059835 (+1139481us) ts_tablet_manager.cc:1145] Deleting tablet 773f2c6e6d7f415ab1218f2365c880a2
0109 20:52:56.059879 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.170960 (+111081us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.215867 (+ 44907us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.313937 (+ 98070us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138360}
I20250109 20:52:56.314375 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 707) took 1392 ms. Trace:
I20250109 20:52:56.314455 1166 rpcz_store.cc:276] 0109 20:52:54.922317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.922356 (+ 39us) service_pool.cc:225] Handling call
0109 20:52:56.314275 (+1391919us) ts_tablet_manager.cc:1145] Deleting tablet 29f1978ebf88470b80b8d4e38c6bc609
0109 20:52:56.314284 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.314361 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1391770}
I20250109 20:52:56.314731 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 708) took 1392 ms. Trace:
I20250109 20:52:56.314802 1166 rpcz_store.cc:276] 0109 20:52:54.922416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.922533 (+ 117us) service_pool.cc:225] Handling call
0109 20:52:56.314644 (+1392111us) ts_tablet_manager.cc:1145] Deleting tablet 219a4b508050490cb3f1ee418e0edbf8
0109 20:52:56.314652 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.314718 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1391995}
I20250109 20:52:56.315017 1166 tablet_replica.cc:331] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.315181 1166 raft_consensus.cc:2238] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.315300 1166 raft_consensus.cc:2267] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.315768 1166 ts_tablet_manager.cc:1905] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.322196 1926 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:56.322309 1032 tablet_service.cc:1467] Processing CreateTablet for tablet ef0f27e5506444fd996d9ded42dfacb7 (DEFAULT_TABLE table=default.Table_15d6655499224a0487b0c1f7feb8e3b9 [id=1c0abe6c795949f0a630b57d06edafd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.322594 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0f27e5506444fd996d9ded42dfacb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.326217 1030 tablet_service.cc:1467] Processing CreateTablet for tablet cbbdc5198b314511abfa78d8c8727e18 (DEFAULT_TABLE table=default.Table_e6a87107149f421e93c2ef7e26400ef1 [id=7878e437152b4d90b49864428cd94ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.326320 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 773f2c6e6d7f415ab1218f2365c880a2 (table default.Table_67d85bf32e4c4a9283bf48b7265d32cc [id=f61e7aec1ea74c409648bee9f041ee0f]) successfully deleted
I20250109 20:52:56.326512 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbdc5198b314511abfa78d8c8727e18. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:56.326762 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 219a4b508050490cb3f1ee418e0edbf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 219a4b508050490cb3f1ee418e0edbf8 already in progress: creating tablet
W20250109 20:52:56.327304 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d4c0f80ec2094da0893fb0a92223b1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c0f80ec2094da0893fb0a92223b1cd already in progress: creating tablet
I20250109 20:52:56.327600 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3464ea8be70f494183c1395c9b2b0784 with delete_type TABLET_DATA_DELETED (replaced by tablet e6dba2d7cf654d449a1c9ff85ac1556f) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.327911 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 530f8e7169f14c84ac2b29314dcc9182 (DEFAULT_TABLE table=default.Table_94c3bc135f0e4e028b93205980b55227 [id=0ef308c37e3647c5917d496cf3151f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.328156 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530f8e7169f14c84ac2b29314dcc9182. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.329195 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec7aec43b744231aa834a14e9d48986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.330597 1018 tablet_service.cc:1467] Processing CreateTablet for tablet ece9f0afb8c24a1f95409d21d052a57b (DEFAULT_TABLE table=default.Table_534986b560ef4cbdaf8e21e045a76d29 [id=91e6e240adc342f1a8a20b0cfb817c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.330857 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece9f0afb8c24a1f95409d21d052a57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.332242 1960 raft_consensus.cc:491] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.332337 1960 raft_consensus.cc:513] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.332934 1960 leader_election.cc:290] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:56.333199 1034 tablet_service.cc:1467] Processing CreateTablet for tablet aec7aec43b744231aa834a14e9d48986 (DEFAULT_TABLE table=default.Table_9fbed746bea74b4086ef2f2dc12674b2 [id=6819b1e50fa34779abd7eb3af5bb49cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.333451 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec7aec43b744231aa834a14e9d48986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.334587 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ed00f7954143c9ba9bf65d8a8a68d2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:56.335307 989 leader_election.cc:343] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.337108 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5cef087cc349eba40c171b326a1f8f (DEFAULT_TABLE table=default.Table_33b6f13c94484781be39523332e10aed [id=9ff0258807514dd790a2cdc5b9bf9be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.337365 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5cef087cc349eba40c171b326a1f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.339390 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6164291d37254e5eb5db712e4a88a3c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.339970 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6164291d37254e5eb5db712e4a88a3c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.339383 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_107c6d6e3acc430f94405d1287423d77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:56.341059 1020 tablet_service.cc:1467] Processing CreateTablet for tablet b6b23623677b4e088c14af925b4e07d4 (DEFAULT_TABLE table=default.Table_70499a7567fa4d6498705309cde3b257 [id=3ae741f5fe4343188b6f0d10b2a09b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.341342 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b23623677b4e088c14af925b4e07d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:56.341440 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_107c6d6e3acc430f94405d1287423d77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.344180 887 tablet_service.cc:1467] Processing CreateTablet for tablet 7075e2a8cf0440b788135f15a182337a (DEFAULT_TABLE table=default.Table_fe23a0c1fdf142f5a7297a0f35955a50 [id=d4f3749d83fc4bc4b9fde5fd8eeaa012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.344770 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ebf92383a5d419ba3e24ea4eea7ee72 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:56.344980 1019 tablet_service.cc:1467] Processing CreateTablet for tablet d8bc15fecd8542e49ff5a0eb22211ec9 (DEFAULT_TABLE table=default.Table_d4091d8d942a404cbea7396af4558981 [id=d792d447bf93470bb29ae0b9423783ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.345284 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bc15fecd8542e49ff5a0eb22211ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.345692 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5c4d17cdde7b4233916baefff36aeb42" } modify_external_catalogs: true
I20250109 20:52:56.345849 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5c4d17cdde7b4233916baefff36aeb42" } modify_external_catalogs: true
I20250109 20:52:56.347004 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5ea5b7134f624208ae0a0e09bbbec68e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.347443 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ea5b7134f624208ae0a0e09bbbec68e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.348779 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 7075e2a8cf0440b788135f15a182337a (DEFAULT_TABLE table=default.Table_fe23a0c1fdf142f5a7297a0f35955a50 [id=d4f3749d83fc4bc4b9fde5fd8eeaa012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.349066 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7075e2a8cf0440b788135f15a182337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.349428 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbdc5198b314511abfa78d8c8727e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.353169 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f5947674a4d6ea08cc52e72f905f0 (DEFAULT_TABLE table=default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.353477 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7f5947674a4d6ea08cc52e72f905f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.354344 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7075e2a8cf0440b788135f15a182337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.358747 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7592f0da4beb49cda4e5e9c6392a085b" } modify_external_catalogs: true
I20250109 20:52:56.358923 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7592f0da4beb49cda4e5e9c6392a085b" } modify_external_catalogs: true
I20250109 20:52:56.365836 1165 tablet_bootstrap.cc:492] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:56.365953 1165 ts_tablet_manager.cc:1397] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.417s user 0.021s sys 0.007s
I20250109 20:52:56.366168 750 tablet_service.cc:1467] Processing CreateTablet for tablet 7075e2a8cf0440b788135f15a182337a (DEFAULT_TABLE table=default.Table_fe23a0c1fdf142f5a7297a0f35955a50 [id=d4f3749d83fc4bc4b9fde5fd8eeaa012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.366551 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7075e2a8cf0440b788135f15a182337a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.367890 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c24377e7b6354110a36711c69a878c58
I20250109 20:52:56.368149 743 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0fdf7cada746ea91e8fb9debcc9475 (DEFAULT_TABLE table=default.Table_0915efe95c4141fa8a554a6b8fb28fea [id=8248bd58bbfd40818738ff7cb4decb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.368312 754 tablet_service.cc:1467] Processing CreateTablet for tablet 2495dc49929f45dbb55d2e285d0264df (DEFAULT_TABLE table=default.Table_a2537a166da14ae39e657d27dc555f71 [id=ddaa5387fa2a45f1ac5650dea9e30724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.369381 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c24377e7b6354110a36711c69a878c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.369979 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c24377e7b6354110a36711c69a878c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c24377e7b6354110a36711c69a878c58 already in progress: creating tablet
I20250109 20:52:56.370329 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2495dc49929f45dbb55d2e285d0264df (DEFAULT_TABLE table=default.Table_a2537a166da14ae39e657d27dc555f71 [id=ddaa5387fa2a45f1ac5650dea9e30724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.370921 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5cef087cc349eba40c171b326a1f8f (DEFAULT_TABLE table=default.Table_33b6f13c94484781be39523332e10aed [id=9ff0258807514dd790a2cdc5b9bf9be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.370671 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2495dc49929f45dbb55d2e285d0264df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.371186 746 tablet_service.cc:1467] Processing CreateTablet for tablet d8bc15fecd8542e49ff5a0eb22211ec9 (DEFAULT_TABLE table=default.Table_d4091d8d942a404cbea7396af4558981 [id=d792d447bf93470bb29ae0b9423783ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.371369 747 tablet_service.cc:1467] Processing CreateTablet for tablet ece9f0afb8c24a1f95409d21d052a57b (DEFAULT_TABLE table=default.Table_534986b560ef4cbdaf8e21e045a76d29 [id=91e6e240adc342f1a8a20b0cfb817c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.371574 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0fdf7cada746ea91e8fb9debcc9475. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.365164 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ed00f7954143c9ba9bf65d8a8a68d2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:56.373584 757 tablet_service.cc:1467] Processing CreateTablet for tablet ef0f27e5506444fd996d9ded42dfacb7 (DEFAULT_TABLE table=default.Table_15d6655499224a0487b0c1f7feb8e3b9 [id=1c0abe6c795949f0a630b57d06edafd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.374329 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_7336cda1ce164722949bdefb9f26e7cd [id=17d12fac83d6463fb91fab6729a08139] alter complete (version 2)
I20250109 20:52:56.374595 739 tablet_service.cc:1467] Processing CreateTablet for tablet 1c422468931d4ec18c7d8cf9c392d3a6 (DEFAULT_TABLE table=default.Table_3a97d64313b6410788dafa912c301267 [id=1977e16013a542e087d46f0bc6abf83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.374945 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c422468931d4ec18c7d8cf9c392d3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.375186 740 tablet_service.cc:1467] Processing CreateTablet for tablet cbbdc5198b314511abfa78d8c8727e18 (DEFAULT_TABLE table=default.Table_e6a87107149f421e93c2ef7e26400ef1 [id=7878e437152b4d90b49864428cd94ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.375437 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f03d4e679a1c4793bd83cd979acced88
I20250109 20:52:56.375727 756 tablet_service.cc:1467] Processing CreateTablet for tablet b6b23623677b4e088c14af925b4e07d4 (DEFAULT_TABLE table=default.Table_70499a7567fa4d6498705309cde3b257 [id=3ae741f5fe4343188b6f0d10b2a09b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.376106 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f03d4e679a1c4793bd83cd979acced88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.376107 990 leader_election.cc:343] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.376205 990 leader_election.cc:304] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:56.376567 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f03d4e679a1c4793bd83cd979acced88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f03d4e679a1c4793bd83cd979acced88 already in progress: creating tablet
I20250109 20:52:56.376647 1960 raft_consensus.cc:2743] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.376670 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet c24377e7b6354110a36711c69a878c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.373136 753 tablet_service.cc:1467] Processing CreateTablet for tablet aec7aec43b744231aa834a14e9d48986 (DEFAULT_TABLE table=default.Table_9fbed746bea74b4086ef2f2dc12674b2 [id=6819b1e50fa34779abd7eb3af5bb49cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.378078 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2495dc49929f45dbb55d2e285d0264df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.379467 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b23623677b4e088c14af925b4e07d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.379509 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet f03d4e679a1c4793bd83cd979acced88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.379618 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbdc5198b314511abfa78d8c8727e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.380402 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0f27e5506444fd996d9ded42dfacb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.380932 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b083e6be42f41ecbf23ee57ec17b0a5" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:56.381384 990 leader_election.cc:343] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.381474 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5cef087cc349eba40c171b326a1f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.381482 990 leader_election.cc:304] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.381631 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec7aec43b744231aa834a14e9d48986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.381820 1960 raft_consensus.cc:2743] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.382534 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7f5947674a4d6ea08cc52e72f905f0 (DEFAULT_TABLE table=default.Table_e664b125305d4f36b75af160a7253520 [id=6ef32c20f57344fa8075269276623836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.382658 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ece9f0afb8c24a1f95409d21d052a57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.366499 1165 raft_consensus.cc:357] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:56.382853 1165 raft_consensus.cc:383] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.384594 1165 raft_consensus.cc:738] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.384864 1165 consensus_queue.cc:260] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:56.385097 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bc15fecd8542e49ff5a0eb22211ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.385687 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7f5947674a4d6ea08cc52e72f905f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.386138 1176 tablet_bootstrap.cc:492] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.386250 1176 ts_tablet_manager.cc:1397] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.085s user 0.000s sys 0.002s
I20250109 20:52:56.386758 748 tablet_service.cc:1467] Processing CreateTablet for tablet 530f8e7169f14c84ac2b29314dcc9182 (DEFAULT_TABLE table=default.Table_94c3bc135f0e4e028b93205980b55227 [id=0ef308c37e3647c5917d496cf3151f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.386952 1158 tablet_bootstrap.cc:492] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:56.387058 1158 ts_tablet_manager.cc:1397] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.200s user 0.003s sys 0.014s
I20250109 20:52:56.387179 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530f8e7169f14c84ac2b29314dcc9182. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.390544 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3464ea8be70f494183c1395c9b2b0784 with delete_type TABLET_DATA_DELETED (replaced by tablet e6dba2d7cf654d449a1c9ff85ac1556f) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.391018 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c24377e7b6354110a36711c69a878c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.391124 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_7e98b984f52d4023ac3365b3e23f26e2 [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 5 got 0
I20250109 20:52:56.391180 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f03d4e679a1c4793bd83cd979acced88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.392613 1165 ts_tablet_manager.cc:1428] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250109 20:52:56.395620 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_7e98b984f52d4023ac3365b3e23f26e2 [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 5 got 0
I20250109 20:52:56.396091 1165 tablet_bootstrap.cc:492] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:56.396299 1949 raft_consensus.cc:491] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.396394 1949 raft_consensus.cc:513] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:56.396891 1949 leader_election.cc:290] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:56.397686 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8654c154dab465d8991910698027c3f" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:56.399041 1165 tablet_bootstrap.cc:654] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.399713 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8654c154dab465d8991910698027c3f" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
W20250109 20:52:56.401041 1980 log.cc:926] Time spent T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20250109 20:52:56.400347 854 leader_election.cc:304] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:56.408131 1949 raft_consensus.cc:2743] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.418298 1158 raft_consensus.cc:357] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.420413 1165 tablet_bootstrap.cc:492] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:56.420431 1176 raft_consensus.cc:357] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.420595 1176 raft_consensus.cc:383] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.420639 1176 raft_consensus.cc:738] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.420573 1165 ts_tablet_manager.cc:1397] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250109 20:52:56.421100 1176 consensus_queue.cc:260] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.421324 1165 raft_consensus.cc:357] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:56.421518 1165 raft_consensus.cc:383] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.422672 1925 tablet.cc:1721] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250109 20:52:56.423324 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 44069.8us
I20250109 20:52:56.423743 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_27b5a1135e3b433fb92b1f482abeac58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:56.424507 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_634a6ecf6a7b4731ae4ac6e03c5375d1" } modify_external_catalogs: true
I20250109 20:52:56.424855 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f3548a3fd5ff4cf0867cf1de1bb898c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:56.425128 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_634a6ecf6a7b4731ae4ac6e03c5375d1" } modify_external_catalogs: true
I20250109 20:52:56.421602 1165 raft_consensus.cc:738] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:52:56.425498 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3548a3fd5ff4cf0867cf1de1bb898c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.425501 1165 consensus_queue.cc:260] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:56.425084 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_09eafff3b75242c3822e14ee2d781828" } new_table_name: "default.Table_df4e58889f9d4ef6b81876c48ecf223e" modify_external_catalogs: true
I20250109 20:52:56.424886 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a97de1805a3143398c7a97e52e5d2e64" } modify_external_catalogs: true
I20250109 20:52:56.426789 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a97de1805a3143398c7a97e52e5d2e64" } modify_external_catalogs: true
W20250109 20:52:56.427765 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27b5a1135e3b433fb92b1f482abeac58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:56.431985 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3464ea8be70f494183c1395c9b2b0784 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3464ea8be70f494183c1395c9b2b0784 already in progress: creating tablet
W20250109 20:52:56.432240 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c24377e7b6354110a36711c69a878c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c24377e7b6354110a36711c69a878c58 already in progress: creating tablet
I20250109 20:52:56.418457 1158 raft_consensus.cc:383] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.437330 1158 raft_consensus.cc:738] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.437506 1158 consensus_queue.cc:260] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.438658 1158 ts_tablet_manager.cc:1428] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.052s user 0.001s sys 0.001s
I20250109 20:52:56.438853 1165 ts_tablet_manager.cc:1428] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20250109 20:52:56.443032 1166 ts_tablet_manager.cc:1918] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.443473 1166 log.cc:1198] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/fbe8a9e5b5fe406b988126cd1859eabc
I20250109 20:52:56.443394 1176 ts_tablet_manager.cc:1428] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
I20250109 20:52:56.445179 1166 ts_tablet_manager.cc:1939] T fbe8a9e5b5fe406b988126cd1859eabc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.445453 1165 tablet_bootstrap.cc:492] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:56.418514 1975 raft_consensus.cc:491] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.446477 1975 raft_consensus.cc:513] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.447340 1975 leader_election.cc:290] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.447667 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58d439254f8418091e3f23c8c274240" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:56.447767 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f03d4e679a1c4793bd83cd979acced88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f03d4e679a1c4793bd83cd979acced88 already in progress: creating tablet
W20250109 20:52:56.448308 712 leader_election.cc:343] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.435384 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5c9e541186c64e158515aff3e19ccda9" } modify_external_catalogs: true
I20250109 20:52:56.449745 451 master_service.cc:946] ReplaceTablet: received request to replace tablet a8da9781ab32429297d0c875c1971501 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:56.449784 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5c9e541186c64e158515aff3e19ccda9" } modify_external_catalogs: true
I20250109 20:52:56.450735 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_40bd214d73eb4e41ba9ad8989d56038b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.451241 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40bd214d73eb4e41ba9ad8989d56038b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.439582 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_3585fc2fc449422880a7ebe625c22d09 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 1 got 0
I20250109 20:52:56.452100 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:56.452348 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:56.452527 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:56.452550 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_12fddfe96f694f7ea31a02021492c77d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:56.452862 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]). Expected version 1 got 0
W20250109 20:52:56.453015 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12fddfe96f694f7ea31a02021492c77d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.454027 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_642b497e51524120a0696f61734c20fe" } new_table_name: "default.Table_2b0fdf7d47e64e7eb3de56adffa487a8" modify_external_catalogs: true
I20250109 20:52:56.456936 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d57a8bbb3da84088a98c816102e4e9df
W20250109 20:52:56.457723 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9629fe30752f49478afad9b93dc235eb consensus state has no leader
I20250109 20:52:56.458231 893 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a8bbb3da84088a98c816102e4e9df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.460443 880 tablet_service.cc:1514] Processing DeleteTablet for tablet fbde2eecdaf4470dbc74b0c59f31643c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.460851 1160 tablet_replica.cc:331] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:56.461021 1160 raft_consensus.cc:2238] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.461163 1160 raft_consensus.cc:2267] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.461791 1160 ts_tablet_manager.cc:1905] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.462664 1165 tablet_bootstrap.cc:654] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.467160 1160 ts_tablet_manager.cc:1918] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.467257 1160 log.cc:1198] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/fbde2eecdaf4470dbc74b0c59f31643c
I20250109 20:52:56.468189 1160 ts_tablet_manager.cc:1939] T fbde2eecdaf4470dbc74b0c59f31643c P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:52:56.470443 713 leader_election.cc:343] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: a58d439254f8418091e3f23c8c274240
I20250109 20:52:56.470561 713 leader_election.cc:304] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:56.470948 1975 raft_consensus.cc:2743] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.476646 1158 tablet_bootstrap.cc:492] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:56.478065 1158 tablet_bootstrap.cc:654] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.484494 757 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a8bbb3da84088a98c816102e4e9df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.484768 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a8bbb3da84088a98c816102e4e9df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.485041 757 tablet_service.cc:1514] Processing DeleteTablet for tablet a8da9781ab32429297d0c875c1971501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.485296 1157 tablet_replica.cc:331] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:56.485477 1157 raft_consensus.cc:2238] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.485627 1157 raft_consensus.cc:2267] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.486416 1157 ts_tablet_manager.cc:1905] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:56.486721 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d57a8bbb3da84088a98c816102e4e9df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d57a8bbb3da84088a98c816102e4e9df already in progress: creating tablet
I20250109 20:52:56.487010 755 tablet_service.cc:1514] Processing DeleteTablet for tablet a58d439254f8418091e3f23c8c274240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.489966 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1b296a6b3a0c47eb8dead9e581ac987b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.490126 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d57a8bbb3da84088a98c816102e4e9df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d57a8bbb3da84088a98c816102e4e9df already in progress: creating tablet
I20250109 20:52:56.490409 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet fbde2eecdaf4470dbc74b0c59f31643c (table default.Table_56c8a1eb8409437c856c4af9ece36c61 [id=fd0bcd24e74342f299f6b060b684af79]) successfully deleted
W20250109 20:52:56.490481 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b296a6b3a0c47eb8dead9e581ac987b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.490610 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ae7ad8e7e47a4b3fa7011f6db7700313"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.491099 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae7ad8e7e47a4b3fa7011f6db7700313 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.491871 1157 ts_tablet_manager.cc:1918] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.491947 1157 log.cc:1198] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/a8da9781ab32429297d0c875c1971501
I20250109 20:52:56.491917 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_169be780c136447db26907a86d29867a" } new_table_name: "default.Table_b3c430aa71444401a3fee0a2e1c39582" modify_external_catalogs: true
I20250109 20:52:56.492218 1157 ts_tablet_manager.cc:1939] T a8da9781ab32429297d0c875c1971501 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:56.445469 1176 tablet_bootstrap.cc:492] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.495193 880 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7f0230e20d45acaa48894fa41c1219 (DEFAULT_TABLE table=default.Table_cd252806276c4bc28ada21e4c72c6c08 [id=d341484e994c40b3990935db8ac1c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.495229 753 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7f0230e20d45acaa48894fa41c1219 (DEFAULT_TABLE table=default.Table_cd252806276c4bc28ada21e4c72c6c08 [id=d341484e994c40b3990935db8ac1c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.495568 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7f0230e20d45acaa48894fa41c1219. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.495741 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7f0230e20d45acaa48894fa41c1219. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.496933 1157 tablet_replica.cc:331] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:56.497308 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.498940 1176 tablet_bootstrap.cc:654] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.500048 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_013e0df3d370438496fdffef9ce40df2" } new_table_name: "default.Table_2f6147aaca5c4410a3e4aedb6a916667" modify_external_catalogs: true
I20250109 20:52:56.500262 1975 raft_consensus.cc:491] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.500381 1975 raft_consensus.cc:513] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.500944 1975 leader_election.cc:290] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.501259 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3d0bfb39db40758e82be511d033b4c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.501454 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3d0bfb39db40758e82be511d033b4c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:56.501646 1055 raft_consensus.cc:2463] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
W20250109 20:52:56.501907 712 leader_election.cc:343] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.502027 712 leader_election.cc:304] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.502133 893 tablet_service.cc:1467] Processing CreateTablet for tablet 190ccaff0ca7486f9dac755f2f625677 (DEFAULT_TABLE table=default.Table_b339612e60ae46b48f88dc6d43a1a530 [id=6def774957a2405a8359e4aa856dd1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.502444 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190ccaff0ca7486f9dac755f2f625677. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.502424 1975 raft_consensus.cc:2798] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:56.502490 1975 raft_consensus.cc:491] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:56.502545 1975 raft_consensus.cc:3054] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:56.504901 1157 raft_consensus.cc:2238] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.505092 1157 raft_consensus.cc:2267] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.505609 1960 raft_consensus.cc:491] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.505733 1960 raft_consensus.cc:513] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.506265 1960 leader_election.cc:290] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:56.507289 1165 tablet_bootstrap.cc:492] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:56.507476 1165 ts_tablet_manager.cc:1397] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.004s
I20250109 20:52:56.507441 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f215eee55d4e6a97a384df6ff9d596" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:56.508030 989 leader_election.cc:343] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.505127 1975 raft_consensus.cc:513] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.506757 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f215eee55d4e6a97a384df6ff9d596" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:56.508531 1165 raft_consensus.cc:357] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:56.508715 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 711) took 1545 ms. Trace:
I20250109 20:52:56.508752 1165 raft_consensus.cc:383] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.508823 1165 raft_consensus.cc:738] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.508813 1166 rpcz_store.cc:276] 0109 20:52:54.963043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.963096 (+ 53us) service_pool.cc:225] Handling call
0109 20:52:56.314982 (+1351886us) ts_tablet_manager.cc:1145] Deleting tablet fbe8a9e5b5fe406b988126cd1859eabc
0109 20:52:56.314989 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.318507 (+ 3518us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.442936 (+124429us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.508693 (+ 65757us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1351715}
W20250109 20:52:56.508898 990 leader_election.cc:343] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.508989 990 leader_election.cc:304] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.509030 1165 consensus_queue.cc:260] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:52:56.509680 1960 raft_consensus.cc:2743] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.510609 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3d0bfb39db40758e82be511d033b4c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:56.510757 1055 raft_consensus.cc:3054] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:56.511693 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3d0bfb39db40758e82be511d033b4c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:56.512104 712 leader_election.cc:343] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.513016 1055 raft_consensus.cc:2463] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:56.513072 1166 tablet_replica.cc:331] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.513239 1166 raft_consensus.cc:2238] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.513355 1166 raft_consensus.cc:2267] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.513854 1166 ts_tablet_manager.cc:1905] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.514045 1982 raft_consensus.cc:491] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.514123 713 leader_election.cc:304] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.514153 1982 raft_consensus.cc:513] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.514317 1975 leader_election.cc:290] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.514571 1975 raft_consensus.cc:2798] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:56.514742 1982 leader_election.cc:290] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.515515 1975 raft_consensus.cc:695] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:56.515816 893 tablet_service.cc:1467] Processing CreateTablet for tablet 64073124f28649b0a1fc7234bdd74659 (DEFAULT_TABLE table=default.Table_7202bebfc9474ba2b0ad805191200c6f [id=63f366874ba64fada3af7fa76b32ddbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.515838 1975 consensus_queue.cc:237] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.516219 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64073124f28649b0a1fc7234bdd74659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.516623 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7dc6809d6547f6b0354b64d7149bf4" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.517035 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_10e1ff86f4e8433f8f5174dd6d3f19fb" } new_table_name: "default.Table_eac461c564e24066931cc3bb3d22e313" modify_external_catalogs: true
I20250109 20:52:56.517148 914 raft_consensus.cc:3054] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:56.520344 750 tablet_service.cc:1467] Processing CreateTablet for tablet 190ccaff0ca7486f9dac755f2f625677 (DEFAULT_TABLE table=default.Table_b339612e60ae46b48f88dc6d43a1a530 [id=6def774957a2405a8359e4aa856dd1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.521055 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet fbe8a9e5b5fe406b988126cd1859eabc (table default.Table_d4b677aa33534a4d8261f52878181252 [id=2f89d33acf544985bafb538d400c0d11]) successfully deleted
I20250109 20:52:56.521682 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet a8da9781ab32429297d0c875c1971501 (table default.Table_4c30a9425b844ee187a29bc12cd2704e [id=8fddac7b2a084c15b76824d25240859d]) successfully deleted
I20250109 20:52:56.521107 1157 ts_tablet_manager.cc:1905] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:56.522189 712 leader_election.cc:343] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.525645 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4e74a5b52948adb470c7bbc4616b49 (DEFAULT_TABLE table=default.Table_dc6847928fe0476aae2e436f966abd99 [id=f416597c25be44cb81a484bae1b733f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.525744 755 tablet_service.cc:1467] Processing CreateTablet for tablet 64073124f28649b0a1fc7234bdd74659 (DEFAULT_TABLE table=default.Table_7202bebfc9474ba2b0ad805191200c6f [id=63f366874ba64fada3af7fa76b32ddbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.525995 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4e74a5b52948adb470c7bbc4616b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.515393 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7dc6809d6547f6b0354b64d7149bf4" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:56.526710 1055 raft_consensus.cc:2463] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:56.527122 713 leader_election.cc:304] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.527302 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_894a7c1fbfb54931992683e6feb51276"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.527819 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_894a7c1fbfb54931992683e6feb51276 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.527925 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 64073124f28649b0a1fc7234bdd74659 (DEFAULT_TABLE table=default.Table_7202bebfc9474ba2b0ad805191200c6f [id=63f366874ba64fada3af7fa76b32ddbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.528242 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64073124f28649b0a1fc7234bdd74659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.518668 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7f0230e20d45acaa48894fa41c1219 (DEFAULT_TABLE table=default.Table_cd252806276c4bc28ada21e4c72c6c08 [id=d341484e994c40b3990935db8ac1c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.528966 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7f0230e20d45acaa48894fa41c1219. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:56.528992 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab59617431fb438cb717cccf0e07518b consensus state has no leader
I20250109 20:52:56.529212 1983 raft_consensus.cc:2798] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:56.529275 1983 raft_consensus.cc:491] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:56.529326 1983 raft_consensus.cc:3054] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:56.532048 1983 raft_consensus.cc:513] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.532594 1983 leader_election.cc:290] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.519220 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 190ccaff0ca7486f9dac755f2f625677 (DEFAULT_TABLE table=default.Table_b339612e60ae46b48f88dc6d43a1a530 [id=6def774957a2405a8359e4aa856dd1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.532927 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190ccaff0ca7486f9dac755f2f625677. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.533584 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5e90cd406e394166b3e8dedebf7fd0fc
I20250109 20:52:56.534224 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64073124f28649b0a1fc7234bdd74659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.534657 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 with delete_type TABLET_DATA_DELETED (replaced by tablet ed5ea5d588834f1b8b2b057ec324aadc) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.534916 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7dc6809d6547f6b0354b64d7149bf4" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:56.534931 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e90cd406e394166b3e8dedebf7fd0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.535876 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7dc6809d6547f6b0354b64d7149bf4" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:56.536039 1055 raft_consensus.cc:3054] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:52:56.536242 712 leader_election.cc:343] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.521191 1960 raft_consensus.cc:491] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.536959 1960 raft_consensus.cc:513] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.537151 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 724d15e05a864852a118badae13a5d3d
I20250109 20:52:56.537520 1960 leader_election.cc:290] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:56.537844 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5a6dbb823e4bd3afeb157b3c5771ba" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.537947 1160 tablet_replica.cc:331] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:56.538040 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5a6dbb823e4bd3afeb157b3c5771ba" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:56.538127 1160 raft_consensus.cc:2238] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:56.538412 1160 raft_consensus.cc:2267] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:56.538491 990 leader_election.cc:343] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.538705 1165 ts_tablet_manager.cc:1428] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
W20250109 20:52:56.539296 989 leader_election.cc:343] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.539384 989 leader_election.cc:304] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.539404 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 724d15e05a864852a118badae13a5d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.539445 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f5a5815d9ef54e3e959f151ed08ae812 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5a5815d9ef54e3e959f151ed08ae812 already in progress: creating tablet
I20250109 20:52:56.539806 1055 raft_consensus.cc:2463] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:56.540104 713 leader_election.cc:304] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.540320 1160 ts_tablet_manager.cc:1905] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.540438 1165 tablet_bootstrap.cc:492] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:56.540766 1982 raft_consensus.cc:2798] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:56.540876 1982 raft_consensus.cc:695] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:56.540966 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190ccaff0ca7486f9dac755f2f625677. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.541107 1982 consensus_queue.cc:237] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.541535 1960 raft_consensus.cc:2743] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.548054 1160 ts_tablet_manager.cc:1918] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:56.555441 1157 ts_tablet_manager.cc:1918] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.555538 1157 log.cc:1198] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/a58d439254f8418091e3f23c8c274240
I20250109 20:52:56.555810 1157 ts_tablet_manager.cc:1939] T a58d439254f8418091e3f23c8c274240 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:56.562644 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_24291889d1144434be6778475dccf4a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.563269 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24291889d1144434be6778475dccf4a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.564924 1158 tablet_bootstrap.cc:492] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:56.565097 1158 ts_tablet_manager.cc:1397] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.001s
I20250109 20:52:56.566444 1158 raft_consensus.cc:357] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.566608 1158 raft_consensus.cc:383] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.566653 1158 raft_consensus.cc:738] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.566823 1158 consensus_queue.cc:260] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.567777 1158 ts_tablet_manager.cc:1428] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:56.568125 1158 tablet_bootstrap.cc:492] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:56.569384 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e90cd406e394166b3e8dedebf7fd0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.569572 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 724d15e05a864852a118badae13a5d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.569896 1158 tablet_bootstrap.cc:654] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.570189 757 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4e74a5b52948adb470c7bbc4616b49 (DEFAULT_TABLE table=default.Table_dc6847928fe0476aae2e436f966abd99 [id=f416597c25be44cb81a484bae1b733f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.570574 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4e74a5b52948adb470c7bbc4616b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.571244 1176 tablet_bootstrap.cc:492] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.571344 1176 ts_tablet_manager.cc:1397] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.000s
I20250109 20:52:56.571875 1176 raft_consensus.cc:357] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.572023 1176 raft_consensus.cc:383] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.572058 1176 raft_consensus.cc:738] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.572211 1176 consensus_queue.cc:260] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.572633 1176 ts_tablet_manager.cc:1428] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:52:56.572957 758 tablet_service.cc:1467] Processing CreateTablet for tablet 73b1822cd5744b5197d8141fc66539a0 (DEFAULT_TABLE table=default.Table_4f8710d177654a9c81c383e7aac0841b [id=3a1cd7e71eab4f90ae054b885cee1626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.573256 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b1822cd5744b5197d8141fc66539a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.573078 1176 tablet_bootstrap.cc:492] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.574362 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 724d15e05a864852a118badae13a5d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.574644 1160 log.cc:1198] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/5e90cd406e394166b3e8dedebf7fd0fc
I20250109 20:52:56.575013 1160 ts_tablet_manager.cc:1939] T 5e90cd406e394166b3e8dedebf7fd0fc P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:56.575523 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 5e90cd406e394166b3e8dedebf7fd0fc (table default.Table_a97de1805a3143398c7a97e52e5d2e64 [id=e39eeed4512c4637b813b7ee38d4318d]) successfully deleted
I20250109 20:52:56.553920 889 tablet_service.cc:1467] Processing CreateTablet for tablet 73b1822cd5744b5197d8141fc66539a0 (DEFAULT_TABLE table=default.Table_4f8710d177654a9c81c383e7aac0841b [id=3a1cd7e71eab4f90ae054b885cee1626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.554447 887 tablet_service.cc:1467] Processing CreateTablet for tablet 44cd9b1adc5b48aca7790bfdb08d539e (DEFAULT_TABLE table=default.Table_f34c9c364f74435e80caff9986005108 [id=1687a1e92f0d46679e45485a72fb93a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.555501 885 tablet_service.cc:1467] Processing CreateTablet for tablet ab59617431fb438cb717cccf0e07518b (DEFAULT_TABLE table=default.Table_2b0fdf7d47e64e7eb3de56adffa487a8 [id=875ffa12d3d54c379103b9fb53e0bfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.556787 897 tablet_service.cc:1467] Processing CreateTablet for tablet e034a025bfc84fbaa8aa95c8060ea964 (DEFAULT_TABLE table=default.Table_7420a91ef4554611bfbb139ea53bd075 [id=b27609c7506441ffb3a67b86e3016764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.576102 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44cd9b1adc5b48aca7790bfdb08d539e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:56.576186 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 724d15e05a864852a118badae13a5d3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 724d15e05a864852a118badae13a5d3d already in progress: creating tablet
I20250109 20:52:56.557413 890 tablet_service.cc:1467] Processing CreateTablet for tablet 365d3a2617114ca19de45765a1e95513 (DEFAULT_TABLE table=default.Table_875ee088902c4843a6b6a6403666cd51 [id=49e5be637d3b4257bb1e71e760823e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.558091 896 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbf9224d9674b08b64f9f4e77e040e0 (DEFAULT_TABLE table=default.Table_02714fd933d14554904584c58daa3159 [id=05aa91cc0adf4e36bb8fe15f036b81bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.558722 1165 tablet_bootstrap.cc:654] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.559077 886 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1c9f392439440bba88a86dbd6d853e (DEFAULT_TABLE table=default.Table_51c7449051f04e25bb0ee8eb814de9e1 [id=a31132be8d7d4219a7ea2a9de7b29260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.577312 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365d3a2617114ca19de45765a1e95513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.577486 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1c9f392439440bba88a86dbd6d853e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.577652 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b1822cd5744b5197d8141fc66539a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.577766 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e034a025bfc84fbaa8aa95c8060ea964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.577929 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59617431fb438cb717cccf0e07518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.578774 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbf9224d9674b08b64f9f4e77e040e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.580471 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet a58d439254f8418091e3f23c8c274240 (table default.Table_4c9330ec3e904978bb8b1f0869a0c532 [id=fbf8fa58eaee4147b61be651b770935b]) successfully deleted
I20250109 20:52:56.580587 1166 ts_tablet_manager.cc:1918] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.580677 1166 log.cc:1198] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b405d16cc62444f2ad94fb69bb9fb6ee
I20250109 20:52:56.585624 893 tablet_service.cc:1467] Processing CreateTablet for tablet 33a20cac880641d1aaa87b2472fd7978 (DEFAULT_TABLE table=default.Table_71c13b2579c040b69ad859670b25a737 [id=37712738e1cc4a2687a492914a01cd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.586047 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a20cac880641d1aaa87b2472fd7978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.583050 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 44cd9b1adc5b48aca7790bfdb08d539e (DEFAULT_TABLE table=default.Table_f34c9c364f74435e80caff9986005108 [id=1687a1e92f0d46679e45485a72fb93a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.587442 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4e74a5b52948adb470c7bbc4616b49 (DEFAULT_TABLE table=default.Table_dc6847928fe0476aae2e436f966abd99 [id=f416597c25be44cb81a484bae1b733f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.587659 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44cd9b1adc5b48aca7790bfdb08d539e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.587822 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 73b1822cd5744b5197d8141fc66539a0 (DEFAULT_TABLE table=default.Table_4f8710d177654a9c81c383e7aac0841b [id=3a1cd7e71eab4f90ae054b885cee1626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.588080 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b1822cd5744b5197d8141fc66539a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.588075 756 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbf9224d9674b08b64f9f4e77e040e0 (DEFAULT_TABLE table=default.Table_02714fd933d14554904584c58daa3159 [id=05aa91cc0adf4e36bb8fe15f036b81bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.588238 740 tablet_service.cc:1467] Processing CreateTablet for tablet 44cd9b1adc5b48aca7790bfdb08d539e (DEFAULT_TABLE table=default.Table_f34c9c364f74435e80caff9986005108 [id=1687a1e92f0d46679e45485a72fb93a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.588392 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbf9224d9674b08b64f9f4e77e040e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.588693 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44cd9b1adc5b48aca7790bfdb08d539e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.588945 1023 tablet_service.cc:1467] Processing CreateTablet for tablet ab59617431fb438cb717cccf0e07518b (DEFAULT_TABLE table=default.Table_2b0fdf7d47e64e7eb3de56adffa487a8 [id=875ffa12d3d54c379103b9fb53e0bfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.589219 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59617431fb438cb717cccf0e07518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.590448 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 365d3a2617114ca19de45765a1e95513 (DEFAULT_TABLE table=default.Table_875ee088902c4843a6b6a6403666cd51 [id=49e5be637d3b4257bb1e71e760823e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.590718 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365d3a2617114ca19de45765a1e95513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.592962 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbf9224d9674b08b64f9f4e77e040e0 (DEFAULT_TABLE table=default.Table_02714fd933d14554904584c58daa3159 [id=05aa91cc0adf4e36bb8fe15f036b81bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.593225 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbf9224d9674b08b64f9f4e77e040e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.595049 747 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1c9f392439440bba88a86dbd6d853e (DEFAULT_TABLE table=default.Table_51c7449051f04e25bb0ee8eb814de9e1 [id=a31132be8d7d4219a7ea2a9de7b29260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.595072 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d7539ac3866842c996d4ced88e764877" } modify_external_catalogs: true
I20250109 20:52:56.595288 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d7539ac3866842c996d4ced88e764877" } modify_external_catalogs: true
I20250109 20:52:56.595389 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1c9f392439440bba88a86dbd6d853e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.596555 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5e1520f094e94246921394f3b8b307df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.597051 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1520f094e94246921394f3b8b307df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.597182 1019 tablet_service.cc:1467] Processing CreateTablet for tablet e034a025bfc84fbaa8aa95c8060ea964 (DEFAULT_TABLE table=default.Table_7420a91ef4554611bfbb139ea53bd075 [id=b27609c7506441ffb3a67b86e3016764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.597434 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e034a025bfc84fbaa8aa95c8060ea964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.598202 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9a8af3d5811647918198f9152cac1a8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.598591 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a8af3d5811647918198f9152cac1a8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.599998 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1c9f392439440bba88a86dbd6d853e (DEFAULT_TABLE table=default.Table_51c7449051f04e25bb0ee8eb814de9e1 [id=a31132be8d7d4219a7ea2a9de7b29260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.600286 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1c9f392439440bba88a86dbd6d853e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.600657 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 33a20cac880641d1aaa87b2472fd7978 (DEFAULT_TABLE table=default.Table_71c13b2579c040b69ad859670b25a737 [id=37712738e1cc4a2687a492914a01cd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.600965 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a20cac880641d1aaa87b2472fd7978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.604621 755 tablet_service.cc:1467] Processing CreateTablet for tablet 365d3a2617114ca19de45765a1e95513 (DEFAULT_TABLE table=default.Table_875ee088902c4843a6b6a6403666cd51 [id=49e5be637d3b4257bb1e71e760823e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.599733 758 tablet_service.cc:1467] Processing CreateTablet for tablet e034a025bfc84fbaa8aa95c8060ea964 (DEFAULT_TABLE table=default.Table_7420a91ef4554611bfbb139ea53bd075 [id=b27609c7506441ffb3a67b86e3016764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.605036 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365d3a2617114ca19de45765a1e95513. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.605147 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4e74a5b52948adb470c7bbc4616b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.605921 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e034a025bfc84fbaa8aa95c8060ea964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.616189 1166 ts_tablet_manager.cc:1939] T b405d16cc62444f2ad94fb69bb9fb6ee P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.616931 1158 tablet_bootstrap.cc:492] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:56.617201 1158 ts_tablet_manager.cc:1397] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250109 20:52:56.617916 1158 raft_consensus.cc:357] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.618172 1158 raft_consensus.cc:383] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.618263 1158 raft_consensus.cc:738] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.618515 1158 consensus_queue.cc:260] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.619029 1158 ts_tablet_manager.cc:1428] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:56.619721 1158 tablet_bootstrap.cc:492] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:56.621431 1158 tablet_bootstrap.cc:654] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.622107 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 712) took 1658 ms. Trace:
I20250109 20:52:56.624116 1166 rpcz_store.cc:276] 0109 20:52:54.963145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.963298 (+ 153us) service_pool.cc:225] Handling call
0109 20:52:56.513028 (+1549730us) ts_tablet_manager.cc:1145] Deleting tablet b405d16cc62444f2ad94fb69bb9fb6ee
0109 20:52:56.513038 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.578120 (+ 65082us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.580528 (+ 2408us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.622091 (+ 41563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1549557}
I20250109 20:52:56.629554 1176 tablet_bootstrap.cc:654] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.630686 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 713) took 1658 ms. Trace:
I20250109 20:52:56.630910 1166 rpcz_store.cc:276] 0109 20:52:54.971641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.972526 (+ 885us) service_pool.cc:225] Handling call
0109 20:52:56.626438 (+1653912us) ts_tablet_manager.cc:1145] Deleting tablet 78800ff1dd964d8ba8862e3f4af69336
0109 20:52:56.626445 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.630529 (+ 4084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653747}
I20250109 20:52:56.632702 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 714) took 1660 ms. Trace:
I20250109 20:52:56.633620 1166 rpcz_store.cc:276] 0109 20:52:54.971771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:54.972776 (+ 1005us) service_pool.cc:225] Handling call
0109 20:52:56.631934 (+1659158us) ts_tablet_manager.cc:1145] Deleting tablet 374df8c6d1424697a3b3922f777e9870
0109 20:52:56.631942 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.632481 (+ 539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659029}
I20250109 20:52:56.636457 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 719) took 1520 ms. Trace:
I20250109 20:52:56.636893 1166 rpcz_store.cc:276] 0109 20:52:55.116263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.116540 (+ 277us) service_pool.cc:225] Handling call
0109 20:52:56.635688 (+1519148us) ts_tablet_manager.cc:1145] Deleting tablet 2a4f047e8fa34ca79e78915c61a8ba12
0109 20:52:56.635696 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.636208 (+ 512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518958}
I20250109 20:52:56.638815 757 tablet_service.cc:1467] Processing CreateTablet for tablet 33a20cac880641d1aaa87b2472fd7978 (DEFAULT_TABLE table=default.Table_71c13b2579c040b69ad859670b25a737 [id=37712738e1cc4a2687a492914a01cd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.639544 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 722) took 1492 ms. Trace:
I20250109 20:52:56.639824 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a20cac880641d1aaa87b2472fd7978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.640062 1166 rpcz_store.cc:276] 0109 20:52:55.147123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.147169 (+ 46us) service_pool.cc:225] Handling call
0109 20:52:56.638728 (+1491559us) ts_tablet_manager.cc:1145] Deleting tablet 741d13db17e847039d3f9e25cafc6db0
0109 20:52:56.638894 (+ 166us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.639369 (+ 475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491308}
I20250109 20:52:56.642407 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 723) took 1494 ms. Trace:
I20250109 20:52:56.643007 1166 rpcz_store.cc:276] 0109 20:52:55.148398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.148441 (+ 43us) service_pool.cc:225] Handling call
0109 20:52:56.641869 (+1493428us) ts_tablet_manager.cc:1145] Deleting tablet 3dfd19df44fb463aab5812702cbbbfe4
0109 20:52:56.641880 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.642389 (+ 509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1493057}
W20250109 20:52:56.645323 712 consensus_peers.cc:487] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:56.645259 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 721) took 1498 ms. Trace:
I20250109 20:52:56.646045 1166 rpcz_store.cc:276] 0109 20:52:55.147017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.153050 (+ 6033us) service_pool.cc:225] Handling call
0109 20:52:56.644484 (+1491434us) ts_tablet_manager.cc:1145] Deleting tablet c03310d8c1ff4b3a82dd0ce35992a319
0109 20:52:56.644676 (+ 192us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.645086 (+ 410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491243}
I20250109 20:52:56.649252 1176 tablet_bootstrap.cc:492] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.649502 1176 ts_tablet_manager.cc:1397] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.003s
I20250109 20:52:56.650475 1176 raft_consensus.cc:357] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.651448 1176 raft_consensus.cc:383] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.651580 1176 raft_consensus.cc:738] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.651918 1176 consensus_queue.cc:260] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.652161 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b405d16cc62444f2ad94fb69bb9fb6ee (table default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]) successfully deleted
I20250109 20:52:56.656914 1176 ts_tablet_manager.cc:1428] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20250109 20:52:56.658352 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 78800ff1dd964d8ba8862e3f4af69336 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78800ff1dd964d8ba8862e3f4af69336 already in progress: creating tablet
W20250109 20:52:56.659817 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 374df8c6d1424697a3b3922f777e9870 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 374df8c6d1424697a3b3922f777e9870 already in progress: creating tablet
W20250109 20:52:56.660281 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2a4f047e8fa34ca79e78915c61a8ba12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a4f047e8fa34ca79e78915c61a8ba12 already in progress: creating tablet
W20250109 20:52:56.660825 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 741d13db17e847039d3f9e25cafc6db0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 741d13db17e847039d3f9e25cafc6db0 already in progress: creating tablet
I20250109 20:52:56.648181 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 726) took 1450 ms. Trace:
I20250109 20:52:56.661484 1176 tablet_bootstrap.cc:492] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.661760 1166 rpcz_store.cc:276] 0109 20:52:55.198121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.198175 (+ 54us) service_pool.cc:225] Handling call
0109 20:52:56.647706 (+1449531us) ts_tablet_manager.cc:1145] Deleting tablet 18f85944f66543fe8dd634ad0da0728d
0109 20:52:56.647716 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.648162 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449061}
I20250109 20:52:56.662953 753 tablet_service.cc:1467] Processing CreateTablet for tablet ab59617431fb438cb717cccf0e07518b (DEFAULT_TABLE table=default.Table_2b0fdf7d47e64e7eb3de56adffa487a8 [id=875ffa12d3d54c379103b9fb53e0bfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:56.663589 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 727) took 1461 ms. Trace:
I20250109 20:52:56.664254 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59617431fb438cb717cccf0e07518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:56.664109 1166 rpcz_store.cc:276] 0109 20:52:55.202300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.202641 (+ 341us) service_pool.cc:225] Handling call
0109 20:52:56.663194 (+1460553us) ts_tablet_manager.cc:1145] Deleting tablet 59ebeffe516c4c239ae2ee9040276e53
0109 20:52:56.663203 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.663575 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1460398}
I20250109 20:52:56.664563 1165 tablet_bootstrap.cc:492] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:56.665895 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 728) took 1461 ms. Trace:
I20250109 20:52:56.665671 1165 ts_tablet_manager.cc:1397] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.125s user 0.002s sys 0.006s
I20250109 20:52:56.666185 1166 rpcz_store.cc:276] 0109 20:52:55.204034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.204079 (+ 45us) service_pool.cc:225] Handling call
0109 20:52:56.665324 (+1461245us) ts_tablet_manager.cc:1145] Deleting tablet 7a1d5b0210134ce1a33b5e7603998593
0109 20:52:56.665333 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.665732 (+ 399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1461098}
I20250109 20:52:56.667452 1166 tablet_replica.cc:331] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.668244 1166 raft_consensus.cc:2238] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.669488 1166 raft_consensus.cc:2267] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.670430 1176 tablet_bootstrap.cc:654] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.673447 1166 ts_tablet_manager.cc:1905] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.675828 1158 tablet_bootstrap.cc:492] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:56.676142 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8d1630e4438e4337ba026b41f630a480
W20250109 20:52:56.676214 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 274a58191d00475e8da9ff72dd33524f consensus state has no leader
I20250109 20:52:56.676558 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6164291d37254e5eb5db712e4a88a3c4" } modify_external_catalogs: true
W20250109 20:52:56.676903 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 consensus state has no leader
I20250109 20:52:56.677497 1158 ts_tablet_manager.cc:1397] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.000s
I20250109 20:52:56.677863 448 master_service.cc:946] ReplaceTablet: received request to replace tablet f5a5815d9ef54e3e959f151ed08ae812 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:56.678239 1158 raft_consensus.cc:357] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.678344 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_524731753c064cf5a3b8a7eda94add84" } modify_external_catalogs: true
I20250109 20:52:56.678486 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_524731753c064cf5a3b8a7eda94add84" } modify_external_catalogs: true
I20250109 20:52:56.678532 1158 raft_consensus.cc:383] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.678686 1158 raft_consensus.cc:738] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.679024 1158 consensus_queue.cc:260] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.679556 1158 ts_tablet_manager.cc:1428] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:56.679690 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3b113c9e02ff40f2b63eab79960b09bd" } new_table_name: "default.Table_58817df2014d4f20bc1db1e6d113a63e" modify_external_catalogs: true
I20250109 20:52:56.680905 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_48b3a08a9f7c4555ac5bed1df8590ebb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.681371 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48b3a08a9f7c4555ac5bed1df8590ebb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.681366 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1630e4438e4337ba026b41f630a480 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.682230 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8d1630e4438e4337ba026b41f630a480 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1630e4438e4337ba026b41f630a480 already in progress: creating tablet
I20250109 20:52:56.683089 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6164291d37254e5eb5db712e4a88a3c4" } modify_external_catalogs: true
W20250109 20:52:56.684105 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 165f6304acc849569a8e90d135a8e2d0 consensus state has no leader
I20250109 20:52:56.686712 1165 raft_consensus.cc:357] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.686909 1165 raft_consensus.cc:383] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.686988 1165 raft_consensus.cc:738] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.687211 1165 consensus_queue.cc:260] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.687649 1165 ts_tablet_manager.cc:1428] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.021s user 0.000s sys 0.001s
I20250109 20:52:56.688061 1165 tablet_bootstrap.cc:492] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:56.694927 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e90cd406e394166b3e8dedebf7fd0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.696216 1157 tablet_replica.cc:331] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:56.696445 1157 raft_consensus.cc:2238] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.696727 1157 raft_consensus.cc:2267] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.700763 1176 tablet_bootstrap.cc:492] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.700979 1176 ts_tablet_manager.cc:1397] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20250109 20:52:56.701638 1176 raft_consensus.cc:357] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.701854 1176 raft_consensus.cc:383] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.701937 1176 raft_consensus.cc:738] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.702111 1157 ts_tablet_manager.cc:1905] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.703397 1176 consensus_queue.cc:260] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.704632 1975 raft_consensus.cc:491] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.704905 1975 raft_consensus.cc:513] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.705693 1960 raft_consensus.cc:491] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.705818 1960 raft_consensus.cc:513] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.706282 1960 leader_election.cc:290] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:56.706414 1975 leader_election.cc:290] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.706735 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5ea5d588834f1b8b2b057ec324aadc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:56.707254 990 leader_election.cc:343] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.707338 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62caa775ef1241f68c0323a2d228c8cd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.707826 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62caa775ef1241f68c0323a2d228c8cd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:56.707966 1055 raft_consensus.cc:2463] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:56.708232 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5ea5d588834f1b8b2b057ec324aadc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.708287 713 leader_election.cc:304] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
W20250109 20:52:56.708690 989 leader_election.cc:343] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.708681 1975 raft_consensus.cc:2798] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:56.708793 989 leader_election.cc:304] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.708822 1975 raft_consensus.cc:491] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:56.708909 1975 raft_consensus.cc:3054] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:52:56.709110 712 leader_election.cc:343] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.709208 1960 raft_consensus.cc:2743] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.710366 1176 ts_tablet_manager.cc:1428] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
W20250109 20:52:56.711385 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3dfd19df44fb463aab5812702cbbbfe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dfd19df44fb463aab5812702cbbbfe4 already in progress: creating tablet
W20250109 20:52:56.711647 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c03310d8c1ff4b3a82dd0ce35992a319 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c03310d8c1ff4b3a82dd0ce35992a319 already in progress: creating tablet
W20250109 20:52:56.711855 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 18f85944f66543fe8dd634ad0da0728d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18f85944f66543fe8dd634ad0da0728d already in progress: creating tablet
W20250109 20:52:56.712028 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 59ebeffe516c4c239ae2ee9040276e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59ebeffe516c4c239ae2ee9040276e53 already in progress: creating tablet
W20250109 20:52:56.712210 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7a1d5b0210134ce1a33b5e7603998593 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a1d5b0210134ce1a33b5e7603998593 already in progress: creating tablet
W20250109 20:52:56.712817 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 724d15e05a864852a118badae13a5d3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 724d15e05a864852a118badae13a5d3d already in progress: creating tablet
I20250109 20:52:56.713937 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1630e4438e4337ba026b41f630a480 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.714372 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.714497 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7d7695ff5946abb6ee576431d2e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.714787 1176 tablet_bootstrap.cc:492] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.716236 1176 tablet_bootstrap.cc:654] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.717281 452 catalog_manager.cc:5526] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:56.717744 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:52:56.718621 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1630e4438e4337ba026b41f630a480 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.720034 1166 ts_tablet_manager.cc:1918] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.720168 1166 log.cc:1198] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/66e4bfa188b545349f358adfd0bc714b
I20250109 20:52:56.720446 1166 ts_tablet_manager.cc:1939] T 66e4bfa188b545349f358adfd0bc714b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.721174 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 733) took 1414 ms. Trace:
I20250109 20:52:56.721295 1166 rpcz_store.cc:276] 0109 20:52:55.306326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.329576 (+ 23250us) service_pool.cc:225] Handling call
0109 20:52:56.667276 (+1337700us) ts_tablet_manager.cc:1145] Deleting tablet 66e4bfa188b545349f358adfd0bc714b
0109 20:52:56.667284 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.690594 (+ 23310us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.719949 (+ 29355us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.721159 (+ 1210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337298}
I20250109 20:52:56.725368 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 739) took 1365 ms. Trace:
I20250109 20:52:56.725384 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_613b9c78c2024fb58046e80d8ad29df8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:56.725494 1166 rpcz_store.cc:276] 0109 20:52:55.359728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.359969 (+ 241us) service_pool.cc:225] Handling call
0109 20:52:56.725261 (+1365292us) ts_tablet_manager.cc:1145] Deleting tablet f1534c7240fc49f684af61aef783ee24
0109 20:52:56.725270 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.725355 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365054}
I20250109 20:52:56.725879 1166 tablet_replica.cc:331] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.726042 1166 raft_consensus.cc:2238] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:52:56.726045 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_613b9c78c2024fb58046e80d8ad29df8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.726147 1166 raft_consensus.cc:2267] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.726521 1166 ts_tablet_manager.cc:1905] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.729283 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_261f03425c0e4e77873973fa3746a8c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:56.729627 1165 tablet_bootstrap.cc:654] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:56.729846 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_261f03425c0e4e77873973fa3746a8c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.731050 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.731106 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 66e4bfa188b545349f358adfd0bc714b (table default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]) successfully deleted
W20250109 20:52:56.731310 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f1534c7240fc49f684af61aef783ee24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1534c7240fc49f684af61aef783ee24 already in progress: creating tablet
I20250109 20:52:56.733438 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 47f2f62aab954531ad6eb60b6f4661b8
I20250109 20:52:56.734333 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f2f62aab954531ad6eb60b6f4661b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.734568 1160 tablet_replica.cc:331] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:56.734735 1160 raft_consensus.cc:2238] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:56.735033 1160 raft_consensus.cc:2267] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.735531 1160 ts_tablet_manager.cc:1905] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.736061 1176 tablet_bootstrap.cc:492] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.736162 1176 ts_tablet_manager.cc:1397] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20250109 20:52:56.737118 1176 raft_consensus.cc:357] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.737262 1176 raft_consensus.cc:383] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.737339 1176 raft_consensus.cc:738] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.737517 1176 consensus_queue.cc:260] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.743719 1160 ts_tablet_manager.cc:1918] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:56.744145 1176 ts_tablet_manager.cc:1428] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250109 20:52:56.746834 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f2f62aab954531ad6eb60b6f4661b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.747351 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f2f62aab954531ad6eb60b6f4661b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.748355 1176 tablet_bootstrap.cc:492] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.748909 1160 log.cc:1198] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/47f2f62aab954531ad6eb60b6f4661b8
I20250109 20:52:56.749423 1160 ts_tablet_manager.cc:1939] T 47f2f62aab954531ad6eb60b6f4661b8 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:56.749895 1176 tablet_bootstrap.cc:654] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.749974 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 47f2f62aab954531ad6eb60b6f4661b8 (table default.Table_524731753c064cf5a3b8a7eda94add84 [id=69b26a1bf8564a889d1227cfad55d0e2]) successfully deleted
I20250109 20:52:56.753139 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_881894033dec46b3862c1185df595c40" } new_table_name: "default.Table_09f1cb68ab054ac4836cd848a45b11de" modify_external_catalogs: true
I20250109 20:52:56.755213 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_283db7be437c43d8845b9f6d4d906bc7" } modify_external_catalogs: true
I20250109 20:52:56.755388 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_283db7be437c43d8845b9f6d4d906bc7" } modify_external_catalogs: true
I20250109 20:52:56.756608 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c71e6da6e81e4bb3becf5e6834b8a4ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.757061 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c71e6da6e81e4bb3becf5e6834b8a4ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.758960 1166 ts_tablet_manager.cc:1918] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.759538 1982 consensus_queue.cc:1035] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:56.760848 1166 log.cc:1198] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/705a7c04f3f948e1a5e486d5ccd48ccc
I20250109 20:52:56.761619 1166 ts_tablet_manager.cc:1939] T 705a7c04f3f948e1a5e486d5ccd48ccc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.763731 1165 tablet_bootstrap.cc:492] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:56.763934 1165 ts_tablet_manager.cc:1397] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250109 20:52:56.765103 1165 raft_consensus.cc:357] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.765445 1165 raft_consensus.cc:383] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.765559 1165 raft_consensus.cc:738] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.766036 1165 consensus_queue.cc:260] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.767525 1165 ts_tablet_manager.cc:1428] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:56.768039 1165 tablet_bootstrap.cc:492] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:56.768404 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8e9896750c594aa78160152def6a4f15" } modify_external_catalogs: true
I20250109 20:52:56.768549 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8e9896750c594aa78160152def6a4f15" } modify_external_catalogs: true
I20250109 20:52:56.770174 1176 tablet_bootstrap.cc:492] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.770354 1176 ts_tablet_manager.cc:1397] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250109 20:52:56.771184 1176 raft_consensus.cc:357] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.771396 1176 raft_consensus.cc:383] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.771466 1176 raft_consensus.cc:738] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.771672 1176 consensus_queue.cc:260] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.770515 1157 ts_tablet_manager.cc:1918] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:56.772910 1176 ts_tablet_manager.cc:1428] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:56.772925 1157 log.cc:1198] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/5e90cd406e394166b3e8dedebf7fd0fc
I20250109 20:52:56.773267 1157 ts_tablet_manager.cc:1939] T 5e90cd406e394166b3e8dedebf7fd0fc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:56.773979 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 734) took 1467 ms. Trace:
I20250109 20:52:56.774197 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 5e90cd406e394166b3e8dedebf7fd0fc (table default.Table_a97de1805a3143398c7a97e52e5d2e64 [id=e39eeed4512c4637b813b7ee38d4318d]) successfully deleted
I20250109 20:52:56.774718 1165 tablet_bootstrap.cc:654] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.775048 1166 rpcz_store.cc:276] 0109 20:52:55.306439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.366903 (+ 60464us) service_pool.cc:225] Handling call
0109 20:52:56.725843 (+1358940us) ts_tablet_manager.cc:1145] Deleting tablet 705a7c04f3f948e1a5e486d5ccd48ccc
0109 20:52:56.725851 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.732235 (+ 6384us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.758877 (+ 26642us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.773964 (+ 15087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1358733}
I20250109 20:52:56.777917 1158 tablet_bootstrap.cc:492] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:56.774981 1176 tablet_bootstrap.cc:492] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:56.779307 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 -> Peer 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583): Couldn't send request to peer 36a29ab2002d4eb389cd11778988e1e9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250109 20:52:56.779619 1176 tablet_bootstrap.cc:654] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.780484 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 740) took 1406 ms. Trace:
I20250109 20:52:56.780614 1166 rpcz_store.cc:276] 0109 20:52:55.373802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.376027 (+ 2225us) service_pool.cc:225] Handling call
0109 20:52:56.780390 (+1404363us) ts_tablet_manager.cc:1145] Deleting tablet a792f8c86cd041deb4186424efd0e68a
0109 20:52:56.780397 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.780471 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1404162}
I20250109 20:52:56.781064 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 705a7c04f3f948e1a5e486d5ccd48ccc (table default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]) successfully deleted
I20250109 20:52:56.781180 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 742) took 1402 ms. Trace:
I20250109 20:52:56.781283 1166 rpcz_store.cc:276] 0109 20:52:55.378710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.394573 (+ 15863us) service_pool.cc:225] Handling call
0109 20:52:56.781095 (+1386522us) ts_tablet_manager.cc:1145] Deleting tablet edc803f2936b4538a5e70ef34e5cc5d4
0109 20:52:56.781103 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.781170 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386325}
W20250109 20:52:56.781378 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a792f8c86cd041deb4186424efd0e68a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a792f8c86cd041deb4186424efd0e68a already in progress: creating tablet
I20250109 20:52:56.781708 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 743) took 1374 ms. Trace:
I20250109 20:52:56.781802 1166 rpcz_store.cc:276] 0109 20:52:55.407690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.408866 (+ 1176us) service_pool.cc:225] Handling call
0109 20:52:56.781603 (+1372737us) ts_tablet_manager.cc:1145] Deleting tablet 56fac585b2904ab490a6882dd5252a9a
0109 20:52:56.781610 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.781697 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372561}
W20250109 20:52:56.781886 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3774aacd532242b788a7af6044dde6b3 consensus state has no leader
I20250109 20:52:56.782330 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 745) took 1346 ms. Trace:
I20250109 20:52:56.782521 1166 rpcz_store.cc:276] 0109 20:52:55.436183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.436417 (+ 234us) service_pool.cc:225] Handling call
0109 20:52:56.782220 (+1345803us) ts_tablet_manager.cc:1145] Deleting tablet 33c7eef84a0c46de990aee95ada1e48d
0109 20:52:56.782231 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.782314 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1345639}
I20250109 20:52:56.783124 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 756) took 1263 ms. Trace:
I20250109 20:52:56.783236 1166 rpcz_store.cc:276] 0109 20:52:55.519256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.519301 (+ 45us) service_pool.cc:225] Handling call
0109 20:52:56.783030 (+1263729us) ts_tablet_manager.cc:1145] Deleting tablet 6787f74100894fe6a7053a6d92ea7b96
0109 20:52:56.783040 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.783112 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1263528}
I20250109 20:52:56.783634 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 758) took 1178 ms. Trace:
I20250109 20:52:56.783740 1166 rpcz_store.cc:276] 0109 20:52:55.605406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.607456 (+ 2050us) service_pool.cc:225] Handling call
0109 20:52:56.783543 (+1176087us) ts_tablet_manager.cc:1145] Deleting tablet 0ec1d30b67674facb8b92a2c1a408073
0109 20:52:56.783550 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.783624 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175925}
I20250109 20:52:56.784161 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 759) took 1171 ms. Trace:
I20250109 20:52:56.784296 1166 rpcz_store.cc:276] 0109 20:52:55.613137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.615725 (+ 2588us) service_pool.cc:225] Handling call
0109 20:52:56.784061 (+1168336us) ts_tablet_manager.cc:1145] Deleting tablet 0ee970cc32dd483fa588ff287b7314e5
0109 20:52:56.784069 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.784147 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1168179}
W20250109 20:52:56.785255 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8d1630e4438e4337ba026b41f630a480 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1630e4438e4337ba026b41f630a480 already in progress: creating tablet
I20250109 20:52:56.785279 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 760) took 1154 ms. Trace:
I20250109 20:52:56.785346 1166 rpcz_store.cc:276] 0109 20:52:55.630885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.630954 (+ 69us) service_pool.cc:225] Handling call
0109 20:52:56.785147 (+1154193us) ts_tablet_manager.cc:1145] Deleting tablet 0cede2a1882946a181b18d66f12c5114
0109 20:52:56.785155 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.785267 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1154013}
W20250109 20:52:56.785888 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet edc803f2936b4538a5e70ef34e5cc5d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edc803f2936b4538a5e70ef34e5cc5d4 already in progress: creating tablet
I20250109 20:52:56.785835 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca83a49523b45be9ac7556b40746a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:56.786089 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 56fac585b2904ab490a6882dd5252a9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56fac585b2904ab490a6882dd5252a9a already in progress: creating tablet
I20250109 20:52:56.786293 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 761) took 1149 ms. Trace:
W20250109 20:52:56.786331 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 33c7eef84a0c46de990aee95ada1e48d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c7eef84a0c46de990aee95ada1e48d already in progress: creating tablet
W20250109 20:52:56.786530 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6787f74100894fe6a7053a6d92ea7b96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6787f74100894fe6a7053a6d92ea7b96 already in progress: creating tablet
W20250109 20:52:56.786724 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0ec1d30b67674facb8b92a2c1a408073 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ec1d30b67674facb8b92a2c1a408073 already in progress: creating tablet
W20250109 20:52:56.786963 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0ee970cc32dd483fa588ff287b7314e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ee970cc32dd483fa588ff287b7314e5 already in progress: creating tablet
W20250109 20:52:56.787231 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0cede2a1882946a181b18d66f12c5114 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cede2a1882946a181b18d66f12c5114 already in progress: creating tablet
I20250109 20:52:56.787611 1157 tablet_replica.cc:331] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:56.787921 1158 tablet_bootstrap.cc:654] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.786677 1166 rpcz_store.cc:276] 0109 20:52:55.636984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.640831 (+ 3847us) service_pool.cc:225] Handling call
0109 20:52:56.786218 (+1145387us) ts_tablet_manager.cc:1145] Deleting tablet 2a83e584f7694f5e9407449f73f9c907
0109 20:52:56.786225 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.786281 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1145193}
W20250109 20:52:56.788347 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2a83e584f7694f5e9407449f73f9c907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a83e584f7694f5e9407449f73f9c907 already in progress: creating tablet
W20250109 20:52:56.789058 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 84f3c6c8764a4af1a9111e147c9c6ac3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f3c6c8764a4af1a9111e147c9c6ac3 already in progress: creating tablet
W20250109 20:52:56.789927 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet beca5f1ed9d3434fb7de90cd74a6f09b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet beca5f1ed9d3434fb7de90cd74a6f09b already in progress: creating tablet
I20250109 20:52:56.788004 1157 raft_consensus.cc:2238] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.790527 1157 raft_consensus.cc:2267] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.787628 1975 raft_consensus.cc:513] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.790913 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 60d3f97679c8408b9248b126a2218313 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:56.791189 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 already in progress: creating tablet
I20250109 20:52:56.791807 1176 tablet_bootstrap.cc:492] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.791865 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62caa775ef1241f68c0323a2d228c8cd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:56.791949 1176 ts_tablet_manager.cc:1397] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250109 20:52:56.792194 1975 leader_election.cc:290] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.792238 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62caa775ef1241f68c0323a2d228c8cd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
W20250109 20:52:56.792344 712 leader_election.cc:343] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.792373 1051 raft_consensus.cc:3054] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:56.792685 1157 ts_tablet_manager.cc:1905] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.794602 1051 raft_consensus.cc:2463] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:56.795136 713 leader_election.cc:304] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.795784 1975 raft_consensus.cc:2798] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:56.796334 1975 raft_consensus.cc:695] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:56.796103 1176 raft_consensus.cc:357] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.796618 1176 raft_consensus.cc:383] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.796588 1975 consensus_queue.cc:237] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.796700 1176 raft_consensus.cc:738] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.796922 1176 consensus_queue.cc:260] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.798208 1176 ts_tablet_manager.cc:1428] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:52:56.799436 1176 tablet_bootstrap.cc:492] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.801317 1176 tablet_bootstrap.cc:654] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:56.801719 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c83b61e879fc4979a85c872f2e6ab88c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c83b61e879fc4979a85c872f2e6ab88c already in progress: creating tablet
W20250109 20:52:56.804759 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 92235a34fb1541fda31a05314346d27b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92235a34fb1541fda31a05314346d27b already in progress: creating tablet
I20250109 20:52:56.805111 1166 tablet_replica.cc:331] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.805313 1166 raft_consensus.cc:2238] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.805473 1166 raft_consensus.cc:2267] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.805946 1166 ts_tablet_manager.cc:1905] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.806867 1982 consensus_queue.cc:1035] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:56.809963 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:56.810985 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.816954 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 674c9bc8c66c4f698c340906dee9108a
I20250109 20:52:56.818226 1165 tablet_bootstrap.cc:492] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:56.818471 1165 ts_tablet_manager.cc:1397] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.002s
I20250109 20:52:56.819271 1165 raft_consensus.cc:357] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.823100 1165 raft_consensus.cc:383] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.823778 1165 raft_consensus.cc:738] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.823901 2003 tablet.cc:1721] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:52:56.826833 1960 raft_consensus.cc:491] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.826933 1960 raft_consensus.cc:513] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.827394 1960 leader_election.cc:290] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:56.828411 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7d7695ff5946abb6ee576431d2e87f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:56.828996 990 leader_election.cc:343] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.829746 1960 raft_consensus.cc:491] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.829845 1960 raft_consensus.cc:513] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.830274 1960 leader_election.cc:290] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:56.830765 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f502d666aa043e2a0b67a4e21bcf03e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:56.831256 990 leader_election.cc:343] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:56.833194 712 consensus_peers.cc:487] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:56.837188 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_7e98b984f52d4023ac3365b3e23f26e2 [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 5 got 0
I20250109 20:52:56.838575 1166 ts_tablet_manager.cc:1918] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.838722 1166 log.cc:1198] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/34f215eee55d4e6a97a384df6ff9d596
I20250109 20:52:56.839103 1166 ts_tablet_manager.cc:1939] T 34f215eee55d4e6a97a384df6ff9d596 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.840165 1176 tablet_bootstrap.cc:492] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.840322 1176 ts_tablet_manager.cc:1397] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.002s
I20250109 20:52:56.840956 1176 raft_consensus.cc:357] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.841161 1176 raft_consensus.cc:383] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.841236 1176 raft_consensus.cc:738] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.841451 1176 consensus_queue.cc:260] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.842780 1176 ts_tablet_manager.cc:1428] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:56.843648 1176 tablet_bootstrap.cc:492] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.844116 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_7e98b984f52d4023ac3365b3e23f26e2 [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 5 got 0
I20250109 20:52:56.845124 1176 tablet_bootstrap.cc:654] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.848481 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 34f215eee55d4e6a97a384df6ff9d596 (table default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]) successfully deleted
I20250109 20:52:56.849422 1166 tablet_replica.cc:331] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.849643 1166 raft_consensus.cc:2238] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.849849 1166 raft_consensus.cc:2267] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.850353 1166 ts_tablet_manager.cc:1905] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.827692 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7d7695ff5946abb6ee576431d2e87f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.831907 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f502d666aa043e2a0b67a4e21bcf03e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.845036 1949 raft_consensus.cc:491] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.852465 1949 raft_consensus.cc:513] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.853003 1949 leader_election.cc:290] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.846014 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f5a5815d9ef54e3e959f151ed08ae812 with delete_type TABLET_DATA_DELETED (replaced by tablet ed5ea5d588834f1b8b2b057ec324aadc) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.846637 890 tablet_service.cc:1514] Processing DeleteTablet for tablet e64207928ef141118e7fc8fe7bb2379b with delete_type TABLET_DATA_DELETED (replaced by tablet 2797e1df3d12476f89fb5155523c3935) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.847216 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 773f2c6e6d7f415ab1218f2365c880a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:56.854297 854 leader_election.cc:343] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 773f2c6e6d7f415ab1218f2365c880a2
W20250109 20:52:56.854349 989 leader_election.cc:343] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:56.854439 854 leader_election.cc:343] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 773f2c6e6d7f415ab1218f2365c880a2
I20250109 20:52:56.854441 989 leader_election.cc:304] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.854514 854 leader_election.cc:304] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:52:56.854844 989 leader_election.cc:343] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.854851 1960 raft_consensus.cc:2743] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.854944 989 leader_election.cc:304] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.855113 1960 raft_consensus.cc:2743] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.855309 1160 tablet_replica.cc:331] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:56.855454 1949 raft_consensus.cc:2743] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.855475 1160 raft_consensus.cc:2238] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.855926 1160 raft_consensus.cc:2267] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.856810 1160 ts_tablet_manager.cc:1905] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.857301 1165 consensus_queue.cc:260] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.861990 1160 ts_tablet_manager.cc:1918] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.862133 1160 log.cc:1198] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f5a5815d9ef54e3e959f151ed08ae812
I20250109 20:52:56.862460 1160 ts_tablet_manager.cc:1939] T f5a5815d9ef54e3e959f151ed08ae812 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:56.864825 1160 tablet_replica.cc:331] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:56.865053 1160 raft_consensus.cc:2238] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.865248 1160 raft_consensus.cc:2267] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.865782 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f5a5815d9ef54e3e959f151ed08ae812 (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:52:56.867859 1166 ts_tablet_manager.cc:1918] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.869762 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 95735385d5ab4fe0b29e33bad14eb95c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.870601 1166 log.cc:1198] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/91dbccae0af24a629600a92e7eadfee7
I20250109 20:52:56.871248 1166 ts_tablet_manager.cc:1939] T 91dbccae0af24a629600a92e7eadfee7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.872823 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 91dbccae0af24a629600a92e7eadfee7 (table default.Table_534eb61fef3c42c8bac87ac39a09ef45 [id=0e350177d27e4c8b8bc3cede2166517e]) successfully deleted
I20250109 20:52:56.873436 1166 tablet_replica.cc:331] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.873787 1166 raft_consensus.cc:2238] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.874130 1166 raft_consensus.cc:2267] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.874794 1166 ts_tablet_manager.cc:1905] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.875016 1158 tablet_bootstrap.cc:492] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:56.875181 1158 ts_tablet_manager.cc:1397] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.000s
I20250109 20:52:56.875335 1960 raft_consensus.cc:491] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.875751 1960 raft_consensus.cc:513] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.875828 1158 raft_consensus.cc:357] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.875994 1158 raft_consensus.cc:383] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.876036 1158 raft_consensus.cc:738] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.876292 1960 leader_election.cc:290] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:56.876235 1158 consensus_queue.cc:260] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.876637 1158 ts_tablet_manager.cc:1428] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:52:56.876961 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce64e0f3a004c9387bb592cf19236e7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:56.877449 990 leader_election.cc:343] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.882344 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce64e0f3a004c9387bb592cf19236e7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:56.882851 989 leader_election.cc:343] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.882956 989 leader_election.cc:304] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.883185 1960 raft_consensus.cc:2743] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.885324 1157 ts_tablet_manager.cc:1918] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:56.885445 1157 log.cc:1198] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/47f2f62aab954531ad6eb60b6f4661b8
I20250109 20:52:56.885833 1157 ts_tablet_manager.cc:1939] T 47f2f62aab954531ad6eb60b6f4661b8 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:56.901471 1165 ts_tablet_manager.cc:1428] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.083s user 0.001s sys 0.001s
I20250109 20:52:56.901949 1166 ts_tablet_manager.cc:1918] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.905452 1158 tablet_bootstrap.cc:492] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:56.907879 1160 ts_tablet_manager.cc:1905] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.907999 1166 log.cc:1198] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ed5ea5d588834f1b8b2b057ec324aadc
I20250109 20:52:56.908537 1166 ts_tablet_manager.cc:1939] T ed5ea5d588834f1b8b2b057ec324aadc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.909260 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2972d7dcc5cd4f0e88717b5d94f99f4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.909876 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2972d7dcc5cd4f0e88717b5d94f99f4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.909963 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b16086bc965e4d5c83a477ef56f70482" } modify_external_catalogs: true
I20250109 20:52:56.910137 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b16086bc965e4d5c83a477ef56f70482" } modify_external_catalogs: true
I20250109 20:52:56.911160 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c696e743a5ae4b2ea41b48e15b437c31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.911579 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c696e743a5ae4b2ea41b48e15b437c31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.910929 1165 tablet_bootstrap.cc:492] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:56.911924 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 768) took 1031 ms. Trace:
I20250109 20:52:56.911989 1166 rpcz_store.cc:276] 0109 20:52:55.880124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.880265 (+ 141us) service_pool.cc:225] Handling call
0109 20:52:56.873395 (+993130us) ts_tablet_manager.cc:1145] Deleting tablet ed5ea5d588834f1b8b2b057ec324aadc
0109 20:52:56.873405 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.880071 (+ 6666us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.901857 (+ 21786us) tablet_metadata.cc:641] Metadata flushed
0109 20:52:56.911909 (+ 10052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":992866}
I20250109 20:52:56.912333 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ed5ea5d588834f1b8b2b057ec324aadc (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:52:56.912959 1165 tablet_bootstrap.cc:654] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.914243 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 770) took 1016 ms. Trace:
I20250109 20:52:56.915023 1166 rpcz_store.cc:276] 0109 20:52:55.898078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.898143 (+ 65us) service_pool.cc:225] Handling call
0109 20:52:56.914135 (+1015992us) ts_tablet_manager.cc:1145] Deleting tablet face19003d0042669e3f22553f9fe19a
0109 20:52:56.914144 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.914229 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015811}
W20250109 20:52:56.915871 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet face19003d0042669e3f22553f9fe19a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet face19003d0042669e3f22553f9fe19a already in progress: creating tablet
I20250109 20:52:56.916325 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 771) took 1015 ms. Trace:
I20250109 20:52:56.916911 1166 rpcz_store.cc:276] 0109 20:52:55.900567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.900610 (+ 43us) service_pool.cc:225] Handling call
0109 20:52:56.916232 (+1015622us) ts_tablet_manager.cc:1145] Deleting tablet faeec3dbcaff4de98e94357f20e7e8ad
0109 20:52:56.916240 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.916311 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015440}
W20250109 20:52:56.917747 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet faeec3dbcaff4de98e94357f20e7e8ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faeec3dbcaff4de98e94357f20e7e8ad already in progress: creating tablet
I20250109 20:52:56.918219 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 772) took 1015 ms. Trace:
I20250109 20:52:56.918812 1166 rpcz_store.cc:276] 0109 20:52:55.902686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:55.902729 (+ 43us) service_pool.cc:225] Handling call
0109 20:52:56.918112 (+1015383us) ts_tablet_manager.cc:1145] Deleting tablet 92235a34fb1541fda31a05314346d27b
0109 20:52:56.918123 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:52:56.918202 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015227}
W20250109 20:52:56.919576 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 92235a34fb1541fda31a05314346d27b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92235a34fb1541fda31a05314346d27b already in progress: creating tablet
W20250109 20:52:56.920163 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 57e56f30e6f3450a8426cd40db89e009 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57e56f30e6f3450a8426cd40db89e009 already in progress: creating tablet
W20250109 20:52:56.921027 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 575f3cb9f61143ab949fa85f47edcafb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575f3cb9f61143ab949fa85f47edcafb already in progress: creating tablet
I20250109 20:52:56.921821 1166 tablet_replica.cc:331] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.922315 1166 raft_consensus.cc:2238] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.922930 1166 raft_consensus.cc:2267] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.923874 1166 ts_tablet_manager.cc:1905] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:56.924108 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55bf7303e7354aa99a7b5bb489f442fe consensus state has no leader
I20250109 20:52:56.924429 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a3861f342059441c9d03405107eb97c5" } new_table_name: "default.Table_e3a5454d3c5f44f09a050ae3d20800ae" modify_external_catalogs: true
I20250109 20:52:56.926282 1960 raft_consensus.cc:491] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.926389 1960 raft_consensus.cc:513] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.926874 1960 leader_election.cc:290] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:56.910280 1925 tablet.cc:1721] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:56.927305 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca83a49523b45be9ac7556b40746a83" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:56.928014 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca83a49523b45be9ac7556b40746a83" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:56.928143 990 leader_election.cc:343] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:56.928483 989 leader_election.cc:343] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.928588 989 leader_election.cc:304] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.928997 1960 raft_consensus.cc:2743] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.933384 1975 raft_consensus.cc:491] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.933543 1975 raft_consensus.cc:513] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.924979 2024 tablet.cc:1721] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:56.934516 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b31bd4072941d5aaf8c43dada3ccc7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.935046 1166 ts_tablet_manager.cc:1918] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:56.935030 712 leader_election.cc:343] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.935410 1975 leader_election.cc:290] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.937491 1158 tablet_bootstrap.cc:654] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.935698 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b31bd4072941d5aaf8c43dada3ccc7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:56.937593 1055 raft_consensus.cc:2463] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:56.938190 1166 log.cc:1198] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/47726b58d6714cd1b46848a16a34ff37
I20250109 20:52:56.938282 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 47f2f62aab954531ad6eb60b6f4661b8 (table default.Table_524731753c064cf5a3b8a7eda94add84 [id=69b26a1bf8564a889d1227cfad55d0e2]) successfully deleted
I20250109 20:52:56.938026 713 leader_election.cc:304] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.938863 1975 raft_consensus.cc:2798] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:56.938966 1975 raft_consensus.cc:491] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:56.939062 1975 raft_consensus.cc:3054] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:56.941393 1975 raft_consensus.cc:513] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.942296 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b31bd4072941d5aaf8c43dada3ccc7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:56.942776 712 leader_election.cc:343] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:56.943533 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b31bd4072941d5aaf8c43dada3ccc7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:56.943778 1055 raft_consensus.cc:3054] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:56.946277 1055 raft_consensus.cc:2463] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:56.946637 1975 leader_election.cc:290] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.946684 713 leader_election.cc:304] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:56.947271 1975 raft_consensus.cc:2798] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:56.947507 1176 tablet_bootstrap.cc:492] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:56.947652 1975 raft_consensus.cc:695] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:56.947944 1975 consensus_queue.cc:237] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.948832 1984 consensus_queue.cc:1035] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:56.949894 2003 tablet.cc:1721] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:52:56.953979 2029 tablet.cc:1721] T b31b9785ee21418ba13b426ba94cfa9b P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250109 20:52:56.957767 1176 ts_tablet_manager.cc:1397] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.114s user 0.023s sys 0.000s
I20250109 20:52:56.958761 1176 raft_consensus.cc:357] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.961457 1176 raft_consensus.cc:383] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.961628 1176 raft_consensus.cc:738] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:56.962033 1176 consensus_queue.cc:260] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.963666 1176 ts_tablet_manager.cc:1428] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250109 20:52:56.964932 1176 tablet_bootstrap.cc:492] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:56.966084 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 80faa29a3b314796b68a4073665d2898
W20250109 20:52:56.970440 712 consensus_peers.cc:487] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:56.972221 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 80faa29a3b314796b68a4073665d2898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:56.972584 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 80faa29a3b314796b68a4073665d2898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:56.973002 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 80faa29a3b314796b68a4073665d2898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:56.974191 1166 ts_tablet_manager.cc:1939] T 47726b58d6714cd1b46848a16a34ff37 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:52:56.974934 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 80faa29a3b314796b68a4073665d2898 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80faa29a3b314796b68a4073665d2898 already in progress: creating tablet
I20250109 20:52:56.975234 1983 raft_consensus.cc:491] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:56.975414 1983 raft_consensus.cc:513] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.976138 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 47726b58d6714cd1b46848a16a34ff37 (table default.Table_f33bf5f9431146a5b111b0448a62e85d [id=3123de0f3f7644739ee589a4c7e9edec]) successfully deleted
I20250109 20:52:56.976380 1983 leader_election.cc:290] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:56.976400 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7307ebdb0c794451a6d91d23d69fddc6" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:56.976534 1165 tablet_bootstrap.cc:492] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:56.976644 1165 ts_tablet_manager.cc:1397] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
W20250109 20:52:56.977146 712 leader_election.cc:343] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:56.977661 713 leader_election.cc:343] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 7307ebdb0c794451a6d91d23d69fddc6
I20250109 20:52:56.977749 713 leader_election.cc:304] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:56.978155 1983 raft_consensus.cc:2743] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:56.975165 1176 tablet_bootstrap.cc:654] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:56.979338 1165 raft_consensus.cc:357] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.979543 1165 raft_consensus.cc:383] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:56.979629 1165 raft_consensus.cc:738] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:52:56.979656 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6b7f0230e20d45acaa48894fa41c1219 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b7f0230e20d45acaa48894fa41c1219 already in progress: creating tablet
I20250109 20:52:56.979915 1166 tablet_replica.cc:331] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.980082 1166 raft_consensus.cc:2238] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.980195 1166 raft_consensus.cc:2267] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.980633 1166 ts_tablet_manager.cc:1905] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:56.979900 1165 consensus_queue.cc:260] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:56.982568 1165 ts_tablet_manager.cc:1428] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250109 20:52:56.978761 1160 ts_tablet_manager.cc:1918] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.982820 1160 log.cc:1198] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/e64207928ef141118e7fc8fe7bb2379b
I20250109 20:52:56.983971 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8c82f4521cc94881ad01252c01dd81b8" } new_table_name: "default.Table_fc79e5e4f5114b189904f056b147d163" modify_external_catalogs: true
I20250109 20:52:56.985394 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet eb6379f0b8984b05982788b5e312912f
I20250109 20:52:56.985743 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3757f39963744852aeac877e634f8416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:56.986112 1166 ts_tablet_manager.cc:1918] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:56.986194 1166 log.cc:1198] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/71ac8d15d48c49289712ce27768dc27b
I20250109 20:52:56.986475 1166 ts_tablet_manager.cc:1939] T 71ac8d15d48c49289712ce27768dc27b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:56.986548 1160 ts_tablet_manager.cc:1939] T e64207928ef141118e7fc8fe7bb2379b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:52:56.986824 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3757f39963744852aeac877e634f8416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.986992 1160 tablet_replica.cc:331] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:56.987140 1160 raft_consensus.cc:2238] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.987254 1160 raft_consensus.cc:2267] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.987335 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 71ac8d15d48c49289712ce27768dc27b (table default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]) successfully deleted
I20250109 20:52:56.988237 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e46707392eee49959bca8c498cf57a76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:56.988687 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e46707392eee49959bca8c498cf57a76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:56.989965 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9151edfc060b4199b6ca8ec93a8ab836" } new_table_name: "default.Table_006fed84e3084734b1882e5214e839da" modify_external_catalogs: true
W20250109 20:52:56.991495 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b8d4d4aa418a4831a8f6d0fd432b412c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d4d4aa418a4831a8f6d0fd432b412c already in progress: creating tablet
W20250109 20:52:56.992158 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5ebf92383a5d419ba3e24ea4eea7ee72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ebf92383a5d419ba3e24ea4eea7ee72 already in progress: creating tablet
W20250109 20:52:56.992857 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d4c0f80ec2094da0893fb0a92223b1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c0f80ec2094da0893fb0a92223b1cd already in progress: creating tablet
W20250109 20:52:56.993314 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9af5cc3af3424aa3bd127450d55d6b36 consensus state has no leader
I20250109 20:52:56.993544 451 master_service.cc:946] ReplaceTablet: received request to replace tablet ed5ea5d588834f1b8b2b057ec324aadc from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:56.993279 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet e64207928ef141118e7fc8fe7bb2379b (table default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]) successfully deleted
W20250109 20:52:56.994189 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 49ba8f186a444c8a9cf01d3eb2036c49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49ba8f186a444c8a9cf01d3eb2036c49 already in progress: creating tablet
I20250109 20:52:56.994582 1166 tablet_replica.cc:331] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:56.994767 918 raft_consensus.cc:3054] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:56.994963 1166 raft_consensus.cc:2238] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:56.995117 1166 raft_consensus.cc:2267] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:56.996263 1166 ts_tablet_manager.cc:1905] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.001128 1160 ts_tablet_manager.cc:1905] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.003535 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_29a0e6a6805a4ffa952d1bd55e846a54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.004096 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29a0e6a6805a4ffa952d1bd55e846a54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.006111 1166 ts_tablet_manager.cc:1918] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.006184 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet b31b9785ee21418ba13b426ba94cfa9b (table default.Table_7e98b984f52d4023ac3365b3e23f26e2 [id=4d18944cc25a4da6a21e3040370599e0]). Expected version 5 got 4
I20250109 20:52:57.006220 1166 log.cc:1198] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/a1ed00f7954143c9ba9bf65d8a8a68d2
I20250109 20:52:57.006531 1166 ts_tablet_manager.cc:1939] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.007052 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_213216398c104dca9ffe9a505d0e46fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.008025 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_213216398c104dca9ffe9a505d0e46fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:57.009985 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67be9533983443cbb8fc52031216a973 consensus state has no leader
I20250109 20:52:57.010198 448 master_service.cc:946] ReplaceTablet: received request to replace tablet ab64a4db7fa241d7a0a4fb73622e6eb5 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:57.011891 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 2608f48f5950466e9ab2d12ee87503cb (table default.Table_7336cda1ce164722949bdefb9f26e7cd [id=17d12fac83d6463fb91fab6729a08139]). Expected version 2 got 0
I20250109 20:52:57.012862 1960 raft_consensus.cc:491] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.013001 1960 raft_consensus.cc:513] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.013549 1960 leader_election.cc:290] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:57.014717 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1891cbef8f4b0cae0c15d478db4860" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:57.015188 1165 tablet_bootstrap.cc:492] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:52:57.015333 990 leader_election.cc:343] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.015770 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1891cbef8f4b0cae0c15d478db4860" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:57.016180 989 leader_election.cc:343] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.016259 989 leader_election.cc:304] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:57.016476 1165 tablet_bootstrap.cc:654] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.016669 1960 raft_consensus.cc:2743] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:57.018002 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet a1ed00f7954143c9ba9bf65d8a8a68d2 (table default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]) successfully deleted
W20250109 20:52:57.019146 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3464ea8be70f494183c1395c9b2b0784 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3464ea8be70f494183c1395c9b2b0784 already in progress: creating tablet
W20250109 20:52:57.020972 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c24377e7b6354110a36711c69a878c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c24377e7b6354110a36711c69a878c58 already in progress: creating tablet
W20250109 20:52:57.021683 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f03d4e679a1c4793bd83cd979acced88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f03d4e679a1c4793bd83cd979acced88 already in progress: creating tablet
W20250109 20:52:57.022379 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d57a8bbb3da84088a98c816102e4e9df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d57a8bbb3da84088a98c816102e4e9df already in progress: creating tablet
I20250109 20:52:57.022850 1166 tablet_replica.cc:331] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.023000 1158 tablet_bootstrap.cc:492] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.023203 1166 raft_consensus.cc:2238] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.023334 1166 raft_consensus.cc:2267] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.023306 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1ee95fba580446b8926cbc779fca720f" } modify_external_catalogs: true
I20250109 20:52:57.023485 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1ee95fba580446b8926cbc779fca720f" } modify_external_catalogs: true
I20250109 20:52:57.023757 1166 ts_tablet_manager.cc:1905] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.024264 1158 ts_tablet_manager.cc:1397] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.119s user 0.008s sys 0.000s
I20250109 20:52:57.025144 1158 raft_consensus.cc:357] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.025432 1158 raft_consensus.cc:383] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.025532 1158 raft_consensus.cc:738] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.025805 1158 consensus_queue.cc:260] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:57.028244 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b920fbb432e64b2ea8891e4ca8966de9 consensus state has no leader
I20250109 20:52:57.026296 1158 ts_tablet_manager.cc:1428] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:57.029027 1158 tablet_bootstrap.cc:492] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:57.031323 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5b967deae0f94c0cbfb8ff72d3fe8e1a" } new_table_name: "default.Table_0192f16726024e458eb0eed5b1b5dfd2" modify_external_catalogs: true
I20250109 20:52:57.033778 1166 ts_tablet_manager.cc:1918] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.033876 1166 log.cc:1198] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/8a5a6dbb823e4bd3afeb157b3c5771ba
I20250109 20:52:57.034166 1166 ts_tablet_manager.cc:1939] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.034768 1158 tablet_bootstrap.cc:654] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.035188 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 8a5a6dbb823e4bd3afeb157b3c5771ba (table default.Table_e677fcf0892746659b8f6d9f5dce6bc1 [id=d8d809ef804f4f909b825f92a823fb8d]) successfully deleted
I20250109 20:52:57.035527 1166 tablet_replica.cc:331] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.035719 1166 raft_consensus.cc:2238] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.035884 1166 raft_consensus.cc:2267] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.036346 1166 ts_tablet_manager.cc:1905] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.038420 1160 ts_tablet_manager.cc:1918] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.038476 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 3d8d67b624984a0bb327d045c55476da from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:57.041903 1166 ts_tablet_manager.cc:1918] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:57.041996 1166 log.cc:1198] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/5e90cd406e394166b3e8dedebf7fd0fc
I20250109 20:52:57.042304 1166 ts_tablet_manager.cc:1939] T 5e90cd406e394166b3e8dedebf7fd0fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.042497 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0ad7f647b70b4047addd805f70288382"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:57.043169 1176 tablet_bootstrap.cc:492] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.043278 1176 ts_tablet_manager.cc:1397] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.012s
I20250109 20:52:57.043889 1176 raft_consensus.cc:357] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.044044 1176 raft_consensus.cc:383] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.044082 1176 raft_consensus.cc:738] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.044234 1176 consensus_queue.cc:260] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.044646 1176 ts_tablet_manager.cc:1428] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20250109 20:52:57.044888 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ad7f647b70b4047addd805f70288382 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.045730 1160 log.cc:1198] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/773f2c6e6d7f415ab1218f2365c880a2
I20250109 20:52:57.046068 1160 ts_tablet_manager.cc:1939] T 773f2c6e6d7f415ab1218f2365c880a2 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:57.046113 1176 tablet_bootstrap.cc:492] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.047116 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 773f2c6e6d7f415ab1218f2365c880a2 (table default.Table_67d85bf32e4c4a9283bf48b7265d32cc [id=f61e7aec1ea74c409648bee9f041ee0f]) successfully deleted
I20250109 20:52:57.047801 1176 tablet_bootstrap.cc:654] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.051702 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_12fddfe96f694f7ea31a02021492c77d" } modify_external_catalogs: true
I20250109 20:52:57.051928 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_12fddfe96f694f7ea31a02021492c77d" } modify_external_catalogs: true
I20250109 20:52:57.052484 1165 tablet_bootstrap.cc:492] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.052631 1165 ts_tablet_manager.cc:1397] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250109 20:52:57.052646 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 5e90cd406e394166b3e8dedebf7fd0fc (table default.Table_a97de1805a3143398c7a97e52e5d2e64 [id=e39eeed4512c4637b813b7ee38d4318d]) successfully deleted
W20250109 20:52:57.053447 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 724d15e05a864852a118badae13a5d3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 724d15e05a864852a118badae13a5d3d already in progress: creating tablet
I20250109 20:52:57.053613 1160 tablet_replica.cc:331] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:57.053829 1160 raft_consensus.cc:2238] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.053855 1166 tablet_replica.cc:331] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.053975 1160 raft_consensus.cc:2267] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.054010 1166 raft_consensus.cc:2238] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.054136 1166 raft_consensus.cc:2267] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.053274 1165 raft_consensus.cc:357] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.054540 1166 ts_tablet_manager.cc:1905] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.054540 1165 raft_consensus.cc:383] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.054606 1165 raft_consensus.cc:738] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.054980 1160 ts_tablet_manager.cc:1905] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.055591 1158 tablet_bootstrap.cc:492] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.055785 1158 ts_tablet_manager.cc:1397] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250109 20:52:57.056464 1158 raft_consensus.cc:357] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.056692 1158 raft_consensus.cc:383] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.056805 1158 raft_consensus.cc:738] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.057057 1158 consensus_queue.cc:260] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.058257 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed] alter complete (version 1)
I20250109 20:52:57.058822 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 617029392e994f44a60c71ba2e340c57 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:57.061920 1176 tablet_bootstrap.cc:492] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.061976 1158 ts_tablet_manager.cc:1428] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:52:57.062103 1176 ts_tablet_manager.cc:1397] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20250109 20:52:57.062839 1176 raft_consensus.cc:357] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.063061 1176 raft_consensus.cc:383] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.063140 1176 raft_consensus.cc:738] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.063364 1176 consensus_queue.cc:260] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.066581 1158 tablet_bootstrap.cc:492] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:57.066613 1176 ts_tablet_manager.cc:1428] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:57.067495 1176 tablet_bootstrap.cc:492] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:57.068387 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0e0f4e3a82a437ebbb16426fffa5e2f has no consensus state
I20250109 20:52:57.068413 1165 consensus_queue.cc:260] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.072729 1165 ts_tablet_manager.cc:1428] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250109 20:52:57.073208 1165 tablet_bootstrap.cc:492] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.073735 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2fb311daf94d4ba582536280f88fd7b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.074276 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fb311daf94d4ba582536280f88fd7b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.074864 1176 tablet_bootstrap.cc:654] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.077873 1166 ts_tablet_manager.cc:1918] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.078008 1166 log.cc:1198] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/8d1630e4438e4337ba026b41f630a480
I20250109 20:52:57.078361 1166 ts_tablet_manager.cc:1939] T 8d1630e4438e4337ba026b41f630a480 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.079396 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 8d1630e4438e4337ba026b41f630a480 (table default.Table_d7539ac3866842c996d4ced88e764877 [id=b78805ca41e647e1927d2564fb00b21f]) successfully deleted
I20250109 20:52:57.079865 1166 tablet_replica.cc:331] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.080129 1158 tablet_bootstrap.cc:654] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.080288 1166 raft_consensus.cc:2238] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.081327 1165 tablet_bootstrap.cc:654] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.081557 1166 raft_consensus.cc:2267] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.082388 1166 ts_tablet_manager.cc:1905] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.086405 1176 tablet_bootstrap.cc:492] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.086545 1176 ts_tablet_manager.cc:1397] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250109 20:52:57.087222 1176 raft_consensus.cc:357] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.087440 1176 raft_consensus.cc:383] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.087522 1176 raft_consensus.cc:738] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.087747 1176 consensus_queue.cc:260] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.091004 1176 ts_tablet_manager.cc:1428] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:57.091584 1960 raft_consensus.cc:491] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.091681 1960 raft_consensus.cc:513] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.092170 1960 leader_election.cc:290] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:57.092639 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f67c4294f8499496db4e82db8c3bf0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:57.092978 451 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
W20250109 20:52:57.093391 990 leader_election.cc:343] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.094451 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f67c4294f8499496db4e82db8c3bf0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:57.094944 989 leader_election.cc:343] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.095052 989 leader_election.cc:304] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:57.095414 1960 raft_consensus.cc:2743] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:57.097846 1176 tablet_bootstrap.cc:492] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.098304 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:57.099699 897 tablet_service.cc:1514] Processing DeleteTablet for tablet deed945f2ecf4290ad002662c3f66030 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.099860 1176 tablet_bootstrap.cc:654] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.105121 1166 ts_tablet_manager.cc:1918] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.106711 1166 log.cc:1198] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/06c3aa20539740cba4e5d6e852ad2909
I20250109 20:52:57.106901 1165 tablet_bootstrap.cc:492] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.107035 1165 ts_tablet_manager.cc:1397] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250109 20:52:57.107576 1166 ts_tablet_manager.cc:1939] T 06c3aa20539740cba4e5d6e852ad2909 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.108069 1165 raft_consensus.cc:357] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.108290 1165 raft_consensus.cc:383] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.108340 1165 raft_consensus.cc:738] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.109251 1158 tablet_bootstrap.cc:492] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.109398 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 06c3aa20539740cba4e5d6e852ad2909 (table default.Table_5e3e729549234c39a601e29da97094d4 [id=5811deb4d7214be490d07cf674920448]) successfully deleted
I20250109 20:52:57.109436 1158 ts_tablet_manager.cc:1397] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.000s
I20250109 20:52:57.110010 1166 tablet_replica.cc:331] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.110169 1158 raft_consensus.cc:357] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.110422 1158 raft_consensus.cc:383] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.110427 1166 raft_consensus.cc:2238] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.110495 1158 raft_consensus.cc:738] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.110741 1158 consensus_queue.cc:260] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.110920 1160 ts_tablet_manager.cc:1918] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.111035 1160 log.cc:1198] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/95735385d5ab4fe0b29e33bad14eb95c
I20250109 20:52:57.111060 1166 raft_consensus.cc:2267] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.111297 1160 ts_tablet_manager.cc:1939] T 95735385d5ab4fe0b29e33bad14eb95c P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:57.112423 1166 ts_tablet_manager.cc:1905] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.112818 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 2b1891cbef8f4b0cae0c15d478db4860 (table default.Table_f20230357dc241449b6b6dc5f5479105 [id=0a0599668d77405ba0e72eda2aa5219e]). Expected version 3 got 0
I20250109 20:52:57.113018 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:57.112968 1176 tablet_bootstrap.cc:492] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.113538 1176 ts_tablet_manager.cc:1397] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20250109 20:52:57.114341 1160 tablet_replica.cc:331] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:57.115235 1158 ts_tablet_manager.cc:1428] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:52:57.117880 1158 tablet_bootstrap.cc:492] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:57.118181 1176 raft_consensus.cc:357] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.118384 1176 raft_consensus.cc:383] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.118448 1176 raft_consensus.cc:738] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.118669 1176 consensus_queue.cc:260] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.120071 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 95735385d5ab4fe0b29e33bad14eb95c (table default.Table_82246d90d1534a7e845106bde0e381f0 [id=a1ff467d633e4d999b85452ff6372219]) successfully deleted
W20250109 20:52:57.120338 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 80faa29a3b314796b68a4073665d2898 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80faa29a3b314796b68a4073665d2898 already in progress: creating tablet
I20250109 20:52:57.120697 1176 ts_tablet_manager.cc:1428] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250109 20:52:57.121115 1176 tablet_bootstrap.cc:492] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.121961 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 42d3318891db42e68836b9fd3b507de4
I20250109 20:52:57.124051 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d3318891db42e68836b9fd3b507de4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:57.125026 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 42d3318891db42e68836b9fd3b507de4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d3318891db42e68836b9fd3b507de4 already in progress: creating tablet
I20250109 20:52:57.125172 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d3318891db42e68836b9fd3b507de4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.125854 1176 tablet_bootstrap.cc:654] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.127554 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a2dba4139826415bae8547cb15dd9863"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.128057 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2dba4139826415bae8547cb15dd9863 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.128230 1158 tablet_bootstrap.cc:654] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.134332 1166 ts_tablet_manager.cc:1918] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.135795 1166 log.cc:1198] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/aa7d7695ff5946abb6ee576431d2e87f
I20250109 20:52:57.136236 1166 ts_tablet_manager.cc:1939] T aa7d7695ff5946abb6ee576431d2e87f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.137498 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet aa7d7695ff5946abb6ee576431d2e87f (table default.Table_d311b482665944949ba9d26a91d7fd39 [id=d1eb369ad1da4221b0be6419fe55ad33]) successfully deleted
I20250109 20:52:57.139096 1166 tablet_replica.cc:331] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.139650 1166 raft_consensus.cc:2238] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.140585 1166 raft_consensus.cc:2267] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.140447 1176 tablet_bootstrap.cc:492] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.140915 1176 ts_tablet_manager.cc:1397] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20250109 20:52:57.141510 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 617029392e994f44a60c71ba2e340c57
I20250109 20:52:57.141541 1176 raft_consensus.cc:357] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.141682 1176 raft_consensus.cc:383] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.141939 454 catalog_manager.cc:6596] ReplaceTablet: tablet 617029392e994f44a60c71ba2e340c57 deleted and replaced by tablet b67810d55d744031a81c177ec07e3cfe
I20250109 20:52:57.142094 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 617029392e994f44a60c71ba2e340c57 with delete_type TABLET_DATA_DELETED (replaced by tablet b67810d55d744031a81c177ec07e3cfe) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.142680 1158 tablet_bootstrap.cc:492] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.142844 1158 ts_tablet_manager.cc:1397] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.004s
W20250109 20:52:57.142972 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 617029392e994f44a60c71ba2e340c57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617029392e994f44a60c71ba2e340c57 already in progress: creating tablet
I20250109 20:52:57.143224 1176 raft_consensus.cc:738] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.142741 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 617029392e994f44a60c71ba2e340c57 with delete_type TABLET_DATA_DELETED (replaced by tablet b67810d55d744031a81c177ec07e3cfe) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.143512 1176 consensus_queue.cc:260] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.143822 1158 raft_consensus.cc:357] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.144779 1158 raft_consensus.cc:383] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.144886 1158 raft_consensus.cc:738] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.145156 1158 consensus_queue.cc:260] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.145649 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c21468e8f0b14e899d210feae2ef211e" } new_table_name: "default.Table_2fbb94fb778f42fb969721dc24cbc33b" modify_external_catalogs: true
I20250109 20:52:57.146173 1176 ts_tablet_manager.cc:1428] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:57.146529 1176 tablet_bootstrap.cc:492] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.148001 1176 tablet_bootstrap.cc:654] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.122905 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d3318891db42e68836b9fd3b507de4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.148869 1166 ts_tablet_manager.cc:1905] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.151788 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 60d3f97679c8408b9248b126a2218313
I20250109 20:52:57.152320 449 catalog_manager.cc:6596] ReplaceTablet: tablet 60d3f97679c8408b9248b126a2218313 deleted and replaced by tablet 94ed3ab88fd8470bbbc27d801323dc9d
I20250109 20:52:57.152680 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d3f97679c8408b9248b126a2218313 with delete_type TABLET_DATA_DELETED (replaced by tablet 94ed3ab88fd8470bbbc27d801323dc9d) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.152837 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d3f97679c8408b9248b126a2218313 with delete_type TABLET_DATA_DELETED (replaced by tablet 94ed3ab88fd8470bbbc27d801323dc9d) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.152977 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d3f97679c8408b9248b126a2218313 with delete_type TABLET_DATA_DELETED (replaced by tablet 94ed3ab88fd8470bbbc27d801323dc9d) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:57.153285 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 60d3f97679c8408b9248b126a2218313 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d3f97679c8408b9248b126a2218313 already in progress: creating tablet
I20250109 20:52:57.127548 1165 consensus_queue.cc:260] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.142563 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 617029392e994f44a60c71ba2e340c57 with delete_type TABLET_DATA_DELETED (replaced by tablet b67810d55d744031a81c177ec07e3cfe) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.153985 1165 ts_tablet_manager.cc:1428] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20250109 20:52:57.155541 1165 tablet_bootstrap.cc:492] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.156059 1160 raft_consensus.cc:2238] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.157388 1160 raft_consensus.cc:2267] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.157848 1166 ts_tablet_manager.cc:1918] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.161218 1160 ts_tablet_manager.cc:1905] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.161710 450 catalog_manager.cc:5526] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:57.162393 450 catalog_manager.cc:5526] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:57.162828 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
I20250109 20:52:57.163044 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3]). Expected version 2 got 0
I20250109 20:52:57.163302 1166 log.cc:1198] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/0f502d666aa043e2a0b67a4e21bcf03e
I20250109 20:52:57.163600 1166 ts_tablet_manager.cc:1939] T 0f502d666aa043e2a0b67a4e21bcf03e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.165269 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 0f502d666aa043e2a0b67a4e21bcf03e (table default.Table_b086cd04a8e249a09d729be46eda67b1 [id=6aa6003be0684dc88b8ed001eed73727]) successfully deleted
I20250109 20:52:57.166193 1176 tablet_bootstrap.cc:492] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.166363 1176 ts_tablet_manager.cc:1397] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20250109 20:52:57.167053 1176 raft_consensus.cc:357] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.167286 1176 raft_consensus.cc:383] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.167377 1176 raft_consensus.cc:738] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.167622 1176 consensus_queue.cc:260] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.168380 1176 ts_tablet_manager.cc:1428] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250109 20:52:57.168522 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7293935c0f346e98ec8117cf1887868 consensus state has no leader
I20250109 20:52:57.168902 1176 tablet_bootstrap.cc:492] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.170624 1166 tablet_replica.cc:331] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.170881 1166 raft_consensus.cc:2238] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.172938 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:52:57.173166 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:57.173341 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:57.173514 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:52:57.173629 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f34c9c364f74435e80caff9986005108" } modify_external_catalogs: true
I20250109 20:52:57.173779 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_3585fc2fc449422880a7ebe625c22d09 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 1 got 0
I20250109 20:52:57.174535 1158 ts_tablet_manager.cc:1428] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250109 20:52:57.173805 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f34c9c364f74435e80caff9986005108" } modify_external_catalogs: true
I20250109 20:52:57.176879 1158 tablet_bootstrap.cc:492] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:57.177305 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a39fac67894a208dd7ec3153b05e4a with delete_type TABLET_DATA_DELETED (replaced by tablet 9629fe30752f49478afad9b93dc235eb) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.177496 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596614521a4a92a71b781b162f70b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.162526 1165 tablet_bootstrap.cc:654] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.165518 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 5757b98100804c688f67e7111db2b622 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f8419d8cc7e475ea43dd263cae87aca) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.178871 897 tablet_service.cc:1467] Processing CreateTablet for tablet cca202d24b504defa676ddd61ec5eb30 (DEFAULT_TABLE table=default.Table_12fddfe96f694f7ea31a02021492c77d [id=fbe01c48ca7d49c5bc3ee4b9b6ca6aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.179674 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca202d24b504defa676ddd61ec5eb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.180404 1158 tablet_bootstrap.cc:654] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.170429 1176 tablet_bootstrap.cc:654] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.182171 1166 raft_consensus.cc:2267] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.182785 1166 ts_tablet_manager.cc:1905] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.190917 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.191304 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e8754625463ea42a92ef65a89214 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a95a31468b431fb4304cf6680016da) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.191521 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.195066 1166 ts_tablet_manager.cc:1918] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:57.195574 1166 log.cc:1198] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/47f2f62aab954531ad6eb60b6f4661b8
I20250109 20:52:57.196074 1166 ts_tablet_manager.cc:1939] T 47f2f62aab954531ad6eb60b6f4661b8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.197330 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 47f2f62aab954531ad6eb60b6f4661b8 (table default.Table_524731753c064cf5a3b8a7eda94add84 [id=69b26a1bf8564a889d1227cfad55d0e2]) successfully deleted
I20250109 20:52:57.198341 1166 tablet_replica.cc:331] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.198643 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cf3f4f1f920642a396c170041d629601" } new_table_name: "default.Table_6f6d795dd7a7434c9f4fb1e068e36849" modify_external_catalogs: true
I20250109 20:52:57.198720 1166 raft_consensus.cc:2238] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.198886 1166 raft_consensus.cc:2267] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.200001 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1326ea6538f74cb28d688c06eda263df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:57.200390 1166 ts_tablet_manager.cc:1905] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:57.200461 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1326ea6538f74cb28d688c06eda263df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.200661 1165 tablet_bootstrap.cc:492] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.200806 1165 ts_tablet_manager.cc:1397] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20250109 20:52:57.201453 1165 raft_consensus.cc:357] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.201629 1165 raft_consensus.cc:383] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.201676 1165 raft_consensus.cc:738] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.201879 1165 consensus_queue.cc:260] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.202382 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 93d409409813420689134773c627833b
I20250109 20:52:57.202841 1165 ts_tablet_manager.cc:1428] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:57.203373 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d409409813420689134773c627833b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.203533 1165 tablet_bootstrap.cc:492] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.203711 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d409409813420689134773c627833b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.203549 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d409409813420689134773c627833b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:57.204191 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 93d409409813420689134773c627833b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d409409813420689134773c627833b already in progress: creating tablet
I20250109 20:52:57.206368 1165 tablet_bootstrap.cc:654] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.207453 753 tablet_service.cc:1467] Processing CreateTablet for tablet cca202d24b504defa676ddd61ec5eb30 (DEFAULT_TABLE table=default.Table_12fddfe96f694f7ea31a02021492c77d [id=fbe01c48ca7d49c5bc3ee4b9b6ca6aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.207815 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca202d24b504defa676ddd61ec5eb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.211798 1166 ts_tablet_manager.cc:1918] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.211901 1166 log.cc:1198] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2ca83a49523b45be9ac7556b40746a83
I20250109 20:52:57.212230 1166 ts_tablet_manager.cc:1939] T 2ca83a49523b45be9ac7556b40746a83 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.213296 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2ca83a49523b45be9ac7556b40746a83 (table default.Table_79f5ff4b60444c1fb9e227b39c89e4f4 [id=523cbdd3d4344d24afca6003e115d5ed]) successfully deleted
I20250109 20:52:57.216583 1166 tablet_replica.cc:331] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.216768 1166 raft_consensus.cc:2238] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.216776 1975 raft_consensus.cc:491] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.216883 1166 raft_consensus.cc:2267] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.216895 1975 raft_consensus.cc:513] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.217422 1975 leader_election.cc:290] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:57.218001 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2b022ad9274c0e9b67560389d8dbed" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:57.218156 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2b022ad9274c0e9b67560389d8dbed" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:57.218312 1051 raft_consensus.cc:2463] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
W20250109 20:52:57.218492 712 leader_election.cc:343] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.218649 713 leader_election.cc:304] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:57.219098 1166 ts_tablet_manager.cc:1905] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.226667 1975 raft_consensus.cc:2798] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:57.226773 1975 raft_consensus.cc:491] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:57.226855 1975 raft_consensus.cc:3054] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:57.228320 1166 ts_tablet_manager.cc:1918] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.230381 1975 raft_consensus.cc:513] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.231376 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2b022ad9274c0e9b67560389d8dbed" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:57.228403 1166 log.cc:1198] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/e0f67c4294f8499496db4e82db8c3bf0
W20250109 20:52:57.231935 712 leader_election.cc:343] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.232373 1166 ts_tablet_manager.cc:1939] T e0f67c4294f8499496db4e82db8c3bf0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.232702 1975 leader_election.cc:290] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:57.232800 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2b022ad9274c0e9b67560389d8dbed" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:57.233170 1051 raft_consensus.cc:3054] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:57.236310 1051 raft_consensus.cc:2463] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:57.236694 1983 raft_consensus.cc:491] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.236833 713 leader_election.cc:304] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:57.236954 1983 raft_consensus.cc:513] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.237555 1984 raft_consensus.cc:2798] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:57.237668 1984 raft_consensus.cc:695] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:57.237865 1984 consensus_queue.cc:237] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.237671 1983 leader_election.cc:290] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:57.239663 713 leader_election.cc:343] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8d1630e4438e4337ba026b41f630a480
I20250109 20:52:57.239626 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1630e4438e4337ba026b41f630a480" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:57.240342 712 leader_election.cc:343] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.240424 712 leader_election.cc:304] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:57.241197 1975 raft_consensus.cc:2743] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:57.242337 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet e0f67c4294f8499496db4e82db8c3bf0 (table default.Table_bf29c4a64f7b4d49af314ad6211982b1 [id=7d02be4e89a7426ea835a9041c394fd4]) successfully deleted
W20250109 20:52:57.243654 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 80faa29a3b314796b68a4073665d2898 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80faa29a3b314796b68a4073665d2898 already in progress: creating tablet
W20250109 20:52:57.244477 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 42d3318891db42e68836b9fd3b507de4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d3318891db42e68836b9fd3b507de4 already in progress: creating tablet
W20250109 20:52:57.245201 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 617029392e994f44a60c71ba2e340c57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617029392e994f44a60c71ba2e340c57 already in progress: creating tablet
W20250109 20:52:57.245992 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 60d3f97679c8408b9248b126a2218313 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d3f97679c8408b9248b126a2218313 already in progress: creating tablet
I20250109 20:52:57.246469 1166 tablet_replica.cc:331] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.246809 1166 raft_consensus.cc:2238] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.247170 1166 raft_consensus.cc:2267] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.248250 1166 ts_tablet_manager.cc:1905] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.255386 1034 tablet_service.cc:1467] Processing CreateTablet for tablet cca202d24b504defa676ddd61ec5eb30 (DEFAULT_TABLE table=default.Table_12fddfe96f694f7ea31a02021492c77d [id=fbe01c48ca7d49c5bc3ee4b9b6ca6aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.255728 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca202d24b504defa676ddd61ec5eb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.256750 753 tablet_service.cc:1467] Processing CreateTablet for tablet 060ae86da08145a89633399698911a03 (DEFAULT_TABLE table=default.Table_40bd214d73eb4e41ba9ad8989d56038b [id=560a7c03985f48de81614934f1f4fbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.256894 1166 ts_tablet_manager.cc:1918] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.256966 1166 log.cc:1198] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/35a39fac67894a208dd7ec3153b05e4a
I20250109 20:52:57.257105 890 tablet_service.cc:1467] Processing CreateTablet for tablet 060ae86da08145a89633399698911a03 (DEFAULT_TABLE table=default.Table_40bd214d73eb4e41ba9ad8989d56038b [id=560a7c03985f48de81614934f1f4fbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.257191 1166 ts_tablet_manager.cc:1939] T 35a39fac67894a208dd7ec3153b05e4a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.257442 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060ae86da08145a89633399698911a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.257481 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b47656fe1ae4c9ba4138bcd37c0e6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.260951 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060ae86da08145a89633399698911a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.262843 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 35a39fac67894a208dd7ec3153b05e4a (table default.Table_df4e58889f9d4ef6b81876c48ecf223e [id=fcb76535fbbe44e29ef627a3ae562ce2]) successfully deleted
I20250109 20:52:57.264041 1166 tablet_replica.cc:331] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.264230 1166 raft_consensus.cc:2238] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.264822 1166 raft_consensus.cc:2267] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.265853 1166 ts_tablet_manager.cc:1905] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.270509 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f48177e873b84ff3bc7fabaa2ac6f672" } new_table_name: "default.Table_3e2af93c0f48485e93d72eb35f08e8a3" modify_external_catalogs: true
I20250109 20:52:57.271977 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_31f8ab4bd8064cbb97ea27b925f31e33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.272459 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31f8ab4bd8064cbb97ea27b925f31e33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.272977 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3219be93dffd45bb89a8ac2f801d567a" } modify_external_catalogs: true
I20250109 20:52:57.273145 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3219be93dffd45bb89a8ac2f801d567a" } modify_external_catalogs: true
I20250109 20:52:57.274428 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ab64a4db7fa241d7a0a4fb73622e6eb5
I20250109 20:52:57.274933 448 catalog_manager.cc:6596] ReplaceTablet: tablet ab64a4db7fa241d7a0a4fb73622e6eb5 deleted and replaced by tablet 4eb65a1eb4cc435385add73ad0c3ca9d
I20250109 20:52:57.275378 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7dbdb0f9160444ca9abff301774ac41d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:57.276093 757 tablet_service.cc:1514] Processing DeleteTablet for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eb65a1eb4cc435385add73ad0c3ca9d) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.276225 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eb65a1eb4cc435385add73ad0c3ca9d) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:57.276295 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dbdb0f9160444ca9abff301774ac41d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.276690 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 with delete_type TABLET_DATA_DELETED (replaced by tablet 4eb65a1eb4cc435385add73ad0c3ca9d) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:57.276793 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab64a4db7fa241d7a0a4fb73622e6eb5 already in progress: creating tablet
I20250109 20:52:57.278647 1166 ts_tablet_manager.cc:1918] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.279964 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0ef997a787824972852640e0fa2bfebb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:57.280305 1166 log.cc:1198] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/cd596614521a4a92a71b781b162f70b6
W20250109 20:52:57.280371 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ef997a787824972852640e0fa2bfebb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.280798 1166 ts_tablet_manager.cc:1939] T cd596614521a4a92a71b781b162f70b6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.282765 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet cd596614521a4a92a71b781b162f70b6 (table default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
I20250109 20:52:57.283365 1166 tablet_replica.cc:331] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.283618 1166 raft_consensus.cc:2238] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.283803 1166 raft_consensus.cc:2267] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.284464 1166 ts_tablet_manager.cc:1905] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.285312 1176 tablet_bootstrap.cc:492] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.285475 1176 ts_tablet_manager.cc:1397] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.117s user 0.024s sys 0.016s
I20250109 20:52:57.286160 1176 raft_consensus.cc:357] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.286377 1176 raft_consensus.cc:383] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.286235 890 tablet_service.cc:1467] Processing CreateTablet for tablet e80959e8c0944167b4ddebf4ebb382c5 (DEFAULT_TABLE table=default.Table_27b5a1135e3b433fb92b1f482abeac58 [id=0832d3ee418f4cd096ac379b027847dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.286454 1176 raft_consensus.cc:738] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.286787 1176 consensus_queue.cc:260] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.287571 1160 ts_tablet_manager.cc:1918] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.288300 1158 tablet_bootstrap.cc:492] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.287734 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80959e8c0944167b4ddebf4ebb382c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.288451 1160 log.cc:1198] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/deed945f2ecf4290ad002662c3f66030
I20250109 20:52:57.288811 1160 ts_tablet_manager.cc:1939] T deed945f2ecf4290ad002662c3f66030 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:57.289263 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 060ae86da08145a89633399698911a03 (DEFAULT_TABLE table=default.Table_40bd214d73eb4e41ba9ad8989d56038b [id=560a7c03985f48de81614934f1f4fbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.289603 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 060ae86da08145a89633399698911a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.292851 1158 ts_tablet_manager.cc:1397] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.116s user 0.011s sys 0.011s
I20250109 20:52:57.293183 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet deed945f2ecf4290ad002662c3f66030 (table default.Table_7c349e16f1b547c2b54536d168ff9e4a [id=339708d383f743e59fade9cec26192f7]) successfully deleted
W20250109 20:52:57.293555 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 42d3318891db42e68836b9fd3b507de4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d3318891db42e68836b9fd3b507de4 already in progress: creating tablet
I20250109 20:52:57.294435 1158 raft_consensus.cc:357] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:57.295091 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 60d3f97679c8408b9248b126a2218313 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60d3f97679c8408b9248b126a2218313 already in progress: creating tablet
W20250109 20:52:57.296979 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 617029392e994f44a60c71ba2e340c57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617029392e994f44a60c71ba2e340c57 already in progress: creating tablet
I20250109 20:52:57.297253 1160 tablet_replica.cc:331] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:57.297463 1160 raft_consensus.cc:2238] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.300412 1176 ts_tablet_manager.cc:1428] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250109 20:52:57.300565 1158 raft_consensus.cc:383] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.302361 1176 tablet_bootstrap.cc:492] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.302572 1160 raft_consensus.cc:2267] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.302690 1158 raft_consensus.cc:738] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:57.302726 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 55bf7303e7354aa99a7b5bb489f442fe (table default.Table_09f1cb68ab054ac4836cd848a45b11de [id=a3ccfed03acd4b2d9fe9bcef1683dc4e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 55bf7303e7354aa99a7b5bb489f442fe consensus state has no leader
I20250109 20:52:57.302347 1983 consensus_queue.cc:1035] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:57.302960 1158 consensus_queue.cc:260] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.298301 757 tablet_service.cc:1467] Processing CreateTablet for tablet e80959e8c0944167b4ddebf4ebb382c5 (DEFAULT_TABLE table=default.Table_27b5a1135e3b433fb92b1f482abeac58 [id=0832d3ee418f4cd096ac379b027847dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.303457 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80959e8c0944167b4ddebf4ebb382c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.303720 1176 tablet_bootstrap.cc:654] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.303747 1160 ts_tablet_manager.cc:1905] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.306077 1158 ts_tablet_manager.cc:1428] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:52:57.306383 1158 tablet_bootstrap.cc:492] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:57.307912 1158 tablet_bootstrap.cc:654] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:57.319111 712 consensus_peers.cc:487] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:57.319943 1176 tablet_bootstrap.cc:492] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.320168 1176 ts_tablet_manager.cc:1397] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20250109 20:52:57.320994 1176 raft_consensus.cc:357] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.323051 1176 raft_consensus.cc:383] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.323159 1176 raft_consensus.cc:738] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.323530 1176 consensus_queue.cc:260] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.324203 1160 ts_tablet_manager.cc:1918] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.324620 1160 log.cc:1198] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/5757b98100804c688f67e7111db2b622
I20250109 20:52:57.325042 1160 ts_tablet_manager.cc:1939] T 5757b98100804c688f67e7111db2b622 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:57.325273 1176 ts_tablet_manager.cc:1428] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:57.325568 1176 tablet_bootstrap.cc:492] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.327219 1176 tablet_bootstrap.cc:654] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.328331 1158 tablet_bootstrap.cc:492] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.328506 1158 ts_tablet_manager.cc:1397] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20250109 20:52:57.329352 1158 raft_consensus.cc:357] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.329584 1158 raft_consensus.cc:383] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.329681 1158 raft_consensus.cc:738] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.330003 1158 consensus_queue.cc:260] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.330777 1158 ts_tablet_manager.cc:1428] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:57.331573 1158 tablet_bootstrap.cc:492] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:57.334245 1034 tablet_service.cc:1467] Processing CreateTablet for tablet e80959e8c0944167b4ddebf4ebb382c5 (DEFAULT_TABLE table=default.Table_27b5a1135e3b433fb92b1f482abeac58 [id=0832d3ee418f4cd096ac379b027847dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.334661 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80959e8c0944167b4ddebf4ebb382c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.334640 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 5757b98100804c688f67e7111db2b622 (table default.Table_38382dc912fa4a28864b23129506ec12 [id=76c6d6ae01dc4932906d907f391de212]) successfully deleted
W20250109 20:52:57.335378 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 93d409409813420689134773c627833b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d409409813420689134773c627833b already in progress: creating tablet
I20250109 20:52:57.336073 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cca202d24b504defa676ddd61ec5eb30
I20250109 20:52:57.336417 1160 tablet_replica.cc:331] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:57.336576 1160 raft_consensus.cc:2238] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.336735 1160 raft_consensus.cc:2267] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.338459 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_baf2e0ecb9594bc3af2f86ba420846d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.338959 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baf2e0ecb9594bc3af2f86ba420846d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.340272 897 tablet_service.cc:1514] Processing DeleteTablet for tablet cca202d24b504defa676ddd61ec5eb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.340453 1160 ts_tablet_manager.cc:1905] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.340623 1158 tablet_bootstrap.cc:654] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.342124 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6e96bb9287564f70abcd0b9f9865e77f" } new_table_name: "default.Table_de7dc1f873f44dcb9e4eb13f73a25657" modify_external_catalogs: true
I20250109 20:52:57.343596 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.343820 1157 tablet_replica.cc:331] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:57.343927 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet cca202d24b504defa676ddd61ec5eb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.343977 1157 raft_consensus.cc:2238] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.344004 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8d67b624984a0bb327d045c55476da with delete_type TABLET_DATA_DELETED (replaced by tablet 687db3e734af47339c2546970a71a539) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.344117 1157 raft_consensus.cc:2267] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.344595 1157 ts_tablet_manager.cc:1905] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.344837 757 tablet_service.cc:1514] Processing DeleteTablet for tablet cca202d24b504defa676ddd61ec5eb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.347146 1982 consensus_queue.cc:1035] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:57.349361 1166 ts_tablet_manager.cc:1918] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.352651 1166 log.cc:1198] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/73e82535780c47ca8b9cd94b013b4b42
I20250109 20:52:57.353396 1166 ts_tablet_manager.cc:1939] T 73e82535780c47ca8b9cd94b013b4b42 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.353943 1157 ts_tablet_manager.cc:1918] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.354111 1157 log.cc:1198] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/7307ebdb0c794451a6d91d23d69fddc6
I20250109 20:52:57.357916 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 73e82535780c47ca8b9cd94b013b4b42 (table default.Table_f7702e93eb894ea5bbf0af322f94c29c [id=0193ea72702947b590c27b63fd70f93f]) successfully deleted
I20250109 20:52:57.354161 757 tablet_service.cc:1467] Processing CreateTablet for tablet cb562bcef6c44710a657db7631ac1196 (DEFAULT_TABLE table=default.Table_f3548a3fd5ff4cf0867cf1de1bb898c6 [id=75d4611914324b10b714e43e89872ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.360095 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb562bcef6c44710a657db7631ac1196. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.361516 1166 tablet_replica.cc:331] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.361824 1166 raft_consensus.cc:2238] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.361974 1166 raft_consensus.cc:2267] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.365226 1166 ts_tablet_manager.cc:1905] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.365648 2032 raft_consensus.cc:491] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.365792 2032 raft_consensus.cc:513] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.366386 2032 leader_election.cc:290] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:57.367241 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2608f48f5950466e9ab2d12ee87503cb" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:57.367619 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2608f48f5950466e9ab2d12ee87503cb" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:57.368440 854 leader_election.cc:304] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:57.368935 2032 raft_consensus.cc:2743] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:57.372452 890 tablet_service.cc:1467] Processing CreateTablet for tablet cb562bcef6c44710a657db7631ac1196 (DEFAULT_TABLE table=default.Table_f3548a3fd5ff4cf0867cf1de1bb898c6 [id=75d4611914324b10b714e43e89872ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.372773 1157 ts_tablet_manager.cc:1939] T 7307ebdb0c794451a6d91d23d69fddc6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:57.373680 1016 tablet_service.cc:1467] Processing CreateTablet for tablet cb562bcef6c44710a657db7631ac1196 (DEFAULT_TABLE table=default.Table_f3548a3fd5ff4cf0867cf1de1bb898c6 [id=75d4611914324b10b714e43e89872ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.373744 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 7307ebdb0c794451a6d91d23d69fddc6 (table default.Table_2e81ac284d7740eb9e743bb21d11a625 [id=6f85f27d20024666b822558de484bc6e]) successfully deleted
I20250109 20:52:57.374117 897 tablet_service.cc:1467] Processing CreateTablet for tablet b6df8eda5c5f4c25a1f7ad7e18f18d2d (DEFAULT_TABLE table=default.Table_107c6d6e3acc430f94405d1287423d77 [id=8eca3e22171244f3b570af5b4ff594ca]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:57.374310 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cca202d24b504defa676ddd61ec5eb30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cca202d24b504defa676ddd61ec5eb30 already in progress: creating tablet
I20250109 20:52:57.375703 757 tablet_service.cc:1467] Processing CreateTablet for tablet b6df8eda5c5f4c25a1f7ad7e18f18d2d (DEFAULT_TABLE table=default.Table_107c6d6e3acc430f94405d1287423d77 [id=8eca3e22171244f3b570af5b4ff594ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.376015 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6df8eda5c5f4c25a1f7ad7e18f18d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.376876 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b6df8eda5c5f4c25a1f7ad7e18f18d2d (DEFAULT_TABLE table=default.Table_107c6d6e3acc430f94405d1287423d77 [id=8eca3e22171244f3b570af5b4ff594ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.377231 893 tablet_service.cc:1467] Processing CreateTablet for tablet c1e75afbcc4e43f4870bd4ae466e6830 (DEFAULT_TABLE table=default.Table_be351430ca734ccd895e178828cb82b0 [id=a4cb9d8a4488428b8bbf8a3275b5eadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.378204 758 tablet_service.cc:1467] Processing CreateTablet for tablet c1e75afbcc4e43f4870bd4ae466e6830 (DEFAULT_TABLE table=default.Table_be351430ca734ccd895e178828cb82b0 [id=a4cb9d8a4488428b8bbf8a3275b5eadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.378477 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e75afbcc4e43f4870bd4ae466e6830. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.381605 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6df8eda5c5f4c25a1f7ad7e18f18d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.381875 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb562bcef6c44710a657db7631ac1196. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.384717 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 44cd9b1adc5b48aca7790bfdb08d539e
I20250109 20:52:57.387521 449 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:52:57.388273 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 44cd9b1adc5b48aca7790bfdb08d539e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.388337 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6df8eda5c5f4c25a1f7ad7e18f18d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.388862 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e75afbcc4e43f4870bd4ae466e6830. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.389865 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 44cd9b1adc5b48aca7790bfdb08d539e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.390910 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 with delete_type TABLET_DATA_DELETED (replaced by tablet 56fac585b2904ab490a6882dd5252a9a) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.393736 449 catalog_manager.cc:5526] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:57.394362 1158 tablet_bootstrap.cc:492] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.394502 1158 ts_tablet_manager.cc:1397] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250109 20:52:57.395272 1158 raft_consensus.cc:357] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.395447 1158 raft_consensus.cc:383] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.395507 1158 raft_consensus.cc:738] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.395700 1158 consensus_queue.cc:260] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.396158 1158 ts_tablet_manager.cc:1428] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:52:57.396413 1158 tablet_bootstrap.cc:492] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:57.394541 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3]). Expected version 2 got 0
I20250109 20:52:57.398468 449 catalog_manager.cc:5526] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:57.398775 1158 tablet_bootstrap.cc:654] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:57.400537 712 consensus_peers.cc:487] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:57.403502 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:57.403816 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:57.404103 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:52:57.404385 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet af8c0bf27e994e6aa4cbedca16febe12 (table default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]). Expected version 1 got 0
I20250109 20:52:57.395871 1166 ts_tablet_manager.cc:1918] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.404685 1166 log.cc:1198] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/59e1e8754625463ea42a92ef65a89214
I20250109 20:52:57.404883 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_3585fc2fc449422880a7ebe625c22d09 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 1 got 0
I20250109 20:52:57.405033 1166 ts_tablet_manager.cc:1939] T 59e1e8754625463ea42a92ef65a89214 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.405279 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 44cd9b1adc5b48aca7790bfdb08d539e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.405949 1166 tablet_replica.cc:331] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:57.405944 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 44cd9b1adc5b48aca7790bfdb08d539e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44cd9b1adc5b48aca7790bfdb08d539e already in progress: creating tablet
I20250109 20:52:57.406123 1166 raft_consensus.cc:2238] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.406248 1166 raft_consensus.cc:2267] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.406355 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 59e1e8754625463ea42a92ef65a89214 (table default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]) successfully deleted
I20250109 20:52:57.413610 1166 ts_tablet_manager.cc:1905] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.422986 1160 ts_tablet_manager.cc:1918] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.423215 1160 log.cc:1198] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2b47656fe1ae4c9ba4138bcd37c0e6db
I20250109 20:52:57.423573 1160 ts_tablet_manager.cc:1939] T 2b47656fe1ae4c9ba4138bcd37c0e6db P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:57.424072 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2b47656fe1ae4c9ba4138bcd37c0e6db (table default.Table_0c3c6e0557ae46a0a8854d04f3d303e0 [id=c24699f5a3e742578b56eb8bf1a4dc97]) successfully deleted
W20250109 20:52:57.424777 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab64a4db7fa241d7a0a4fb73622e6eb5 already in progress: creating tablet
W20250109 20:52:57.425381 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cca202d24b504defa676ddd61ec5eb30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cca202d24b504defa676ddd61ec5eb30 already in progress: creating tablet
W20250109 20:52:57.425957 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 44cd9b1adc5b48aca7790bfdb08d539e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44cd9b1adc5b48aca7790bfdb08d539e already in progress: creating tablet
I20250109 20:52:57.426278 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb562bcef6c44710a657db7631ac1196. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.420646 1166 ts_tablet_manager.cc:1918] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.441399 1166 log.cc:1198] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f1c5a6b9681d4ef6a36e308ab2916423
I20250109 20:52:57.442014 1166 ts_tablet_manager.cc:1939] T f1c5a6b9681d4ef6a36e308ab2916423 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.443647 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f1c5a6b9681d4ef6a36e308ab2916423 (table default.Table_68ffdf1aeefe420ea24aa1102012f77c [id=4d826d307bff4ab2b43a9d8b718055d6]) successfully deleted
W20250109 20:52:57.446117 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 93d409409813420689134773c627833b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d409409813420689134773c627833b already in progress: creating tablet
W20250109 20:52:57.447515 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ab64a4db7fa241d7a0a4fb73622e6eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab64a4db7fa241d7a0a4fb73622e6eb5 already in progress: creating tablet
W20250109 20:52:57.448541 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cca202d24b504defa676ddd61ec5eb30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cca202d24b504defa676ddd61ec5eb30 already in progress: creating tablet
I20250109 20:52:57.449218 1166 tablet_replica.cc:331] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:57.449263 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd92f4c7abea454c8d68c1f5023d5727 consensus state has no leader
I20250109 20:52:57.449805 1166 raft_consensus.cc:2238] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.449960 1166 raft_consensus.cc:2267] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.451442 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9db2524d4be648e4828df22a952c1cb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.452046 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9db2524d4be648e4828df22a952c1cb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.453259 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d6c47101ffbb45b8b62f228bfa7720e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.453668 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6c47101ffbb45b8b62f228bfa7720e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.458637 1166 ts_tablet_manager.cc:1905] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.462548 1034 tablet_service.cc:1467] Processing CreateTablet for tablet c1e75afbcc4e43f4870bd4ae466e6830 (DEFAULT_TABLE table=default.Table_be351430ca734ccd895e178828cb82b0 [id=a4cb9d8a4488428b8bbf8a3275b5eadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.462656 897 tablet_service.cc:1467] Processing CreateTablet for tablet 66cc0ad69bb142bd9b47a1b78451bee6 (DEFAULT_TABLE table=default.Table_5ea5b7134f624208ae0a0e09bbbec68e [id=ebf5ff04e1d5490891605055362a9820]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.462922 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e75afbcc4e43f4870bd4ae466e6830. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.463006 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66cc0ad69bb142bd9b47a1b78451bee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.469463 1165 tablet_bootstrap.cc:492] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.469640 1165 ts_tablet_manager.cc:1397] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.266s user 0.023s sys 0.028s
I20250109 20:52:57.470508 1165 raft_consensus.cc:357] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.470724 1165 raft_consensus.cc:383] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.470814 1165 raft_consensus.cc:738] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.471045 1165 consensus_queue.cc:260] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.469611 1166 ts_tablet_manager.cc:1918] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.474579 1166 log.cc:1198] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/3d8d67b624984a0bb327d045c55476da
I20250109 20:52:57.474896 1166 ts_tablet_manager.cc:1939] T 3d8d67b624984a0bb327d045c55476da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.475951 1166 tablet_replica.cc:331] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.475931 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 3d8d67b624984a0bb327d045c55476da (table default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]) successfully deleted
I20250109 20:52:57.476130 1166 raft_consensus.cc:2238] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.476262 1166 raft_consensus.cc:2267] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:57.476500 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 44cd9b1adc5b48aca7790bfdb08d539e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44cd9b1adc5b48aca7790bfdb08d539e already in progress: creating tablet
I20250109 20:52:57.477046 1165 ts_tablet_manager.cc:1428] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250109 20:52:57.480427 1165 tablet_bootstrap.cc:492] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.481634 757 tablet_service.cc:1467] Processing CreateTablet for tablet 66cc0ad69bb142bd9b47a1b78451bee6 (DEFAULT_TABLE table=default.Table_5ea5b7134f624208ae0a0e09bbbec68e [id=ebf5ff04e1d5490891605055362a9820]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.482008 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66cc0ad69bb142bd9b47a1b78451bee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.482226 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 66cc0ad69bb142bd9b47a1b78451bee6 (DEFAULT_TABLE table=default.Table_5ea5b7134f624208ae0a0e09bbbec68e [id=ebf5ff04e1d5490891605055362a9820]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.482641 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66cc0ad69bb142bd9b47a1b78451bee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.482604 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4c922ad62c0b46fb8a850c8c9969d2a7 (DEFAULT_TABLE table=default.Table_8a8d16516a0548d9a55647e97d9e03d6 [id=61bd784b0d39495d9f3dc3ca5fbf2470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.483996 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4c922ad62c0b46fb8a850c8c9969d2a7 (DEFAULT_TABLE table=default.Table_8a8d16516a0548d9a55647e97d9e03d6 [id=61bd784b0d39495d9f3dc3ca5fbf2470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.484483 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c922ad62c0b46fb8a850c8c9969d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.488196 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c922ad62c0b46fb8a850c8c9969d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.492255 1165 tablet_bootstrap.cc:654] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.492475 1984 raft_consensus.cc:491] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.492659 1984 raft_consensus.cc:513] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.493147 1166 ts_tablet_manager.cc:1905] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.493932 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e4bfa188b545349f358adfd0bc714b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:57.494400 712 leader_election.cc:343] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.495146 1984 leader_election.cc:290] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:57.495558 713 leader_election.cc:343] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 66e4bfa188b545349f358adfd0bc714b
I20250109 20:52:57.495651 713 leader_election.cc:304] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:57.496055 1984 raft_consensus.cc:2743] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:57.498261 1165 tablet_bootstrap.cc:492] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.498399 1165 ts_tablet_manager.cc:1397] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250109 20:52:57.499086 1165 raft_consensus.cc:357] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.499253 1165 raft_consensus.cc:383] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.499311 1165 raft_consensus.cc:738] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.499513 1165 consensus_queue.cc:260] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.500324 1176 tablet_bootstrap.cc:492] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.500532 1176 ts_tablet_manager.cc:1397] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.175s user 0.010s sys 0.017s
I20250109 20:52:57.501271 1176 raft_consensus.cc:357] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.501443 1176 raft_consensus.cc:383] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.501487 1176 raft_consensus.cc:738] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.501658 1176 consensus_queue.cc:260] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.502233 1176 ts_tablet_manager.cc:1428] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:57.503762 1176 tablet_bootstrap.cc:492] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.504982 1165 ts_tablet_manager.cc:1428] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:52:57.505406 1176 tablet_bootstrap.cc:654] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.506464 1165 tablet_bootstrap.cc:492] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.508071 1165 tablet_bootstrap.cc:654] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.508271 918 raft_consensus.cc:3054] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:57.511427 1166 ts_tablet_manager.cc:1918] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.511510 1166 log.cc:1198] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/16c9f7274a7542e89a19bd4afcf89205
I20250109 20:52:57.511763 1166 ts_tablet_manager.cc:1939] T 16c9f7274a7542e89a19bd4afcf89205 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.514814 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4c922ad62c0b46fb8a850c8c9969d2a7 (DEFAULT_TABLE table=default.Table_8a8d16516a0548d9a55647e97d9e03d6 [id=61bd784b0d39495d9f3dc3ca5fbf2470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.515318 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c922ad62c0b46fb8a850c8c9969d2a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.523861 897 tablet_service.cc:1467] Processing CreateTablet for tablet 5e571b3fa5c146b6b5fe2d814ea99ac0 (DEFAULT_TABLE table=default.Table_78e828c9c8034bd88102f0763df9f88d [id=888dafa806f24147a5db25495f80a76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.524065 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 16c9f7274a7542e89a19bd4afcf89205 (table default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]) successfully deleted
I20250109 20:52:57.524253 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e571b3fa5c146b6b5fe2d814ea99ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.529636 757 tablet_service.cc:1467] Processing CreateTablet for tablet 5e571b3fa5c146b6b5fe2d814ea99ac0 (DEFAULT_TABLE table=default.Table_78e828c9c8034bd88102f0763df9f88d [id=888dafa806f24147a5db25495f80a76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.529981 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e571b3fa5c146b6b5fe2d814ea99ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.530264 1176 tablet_bootstrap.cc:492] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.530403 1176 ts_tablet_manager.cc:1397] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250109 20:52:57.531316 1176 raft_consensus.cc:357] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.531536 1176 raft_consensus.cc:383] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.531591 1176 raft_consensus.cc:738] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.531859 1176 consensus_queue.cc:260] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.532474 1176 ts_tablet_manager.cc:1428] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:57.534098 1176 tablet_bootstrap.cc:492] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.535661 1176 tablet_bootstrap.cc:654] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.539400 1158 tablet_bootstrap.cc:492] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.539543 1158 ts_tablet_manager.cc:1397] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.143s user 0.017s sys 0.003s
I20250109 20:52:57.540268 1158 raft_consensus.cc:357] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.540529 1158 raft_consensus.cc:383] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.540604 1158 raft_consensus.cc:738] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.540921 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 5e571b3fa5c146b6b5fe2d814ea99ac0 (DEFAULT_TABLE table=default.Table_78e828c9c8034bd88102f0763df9f88d [id=888dafa806f24147a5db25495f80a76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.540860 1158 consensus_queue.cc:260] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.541424 1158 ts_tablet_manager.cc:1428] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:52:57.541699 1158 tablet_bootstrap.cc:492] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:57.542950 893 tablet_service.cc:1467] Processing CreateTablet for tablet af77e079fe5642d790707406c5588420 (DEFAULT_TABLE table=default.Table_6f7d71e007e84fafa7d6634d3ee5e122 [id=59389c4ca01a45f08d705864669da116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.543195 1158 tablet_bootstrap.cc:654] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.543852 1165 tablet_bootstrap.cc:492] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.543960 1165 ts_tablet_manager.cc:1397] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20250109 20:52:57.544135 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e571b3fa5c146b6b5fe2d814ea99ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.544634 1165 raft_consensus.cc:357] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.544838 1165 raft_consensus.cc:383] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.544889 1165 raft_consensus.cc:738] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.545085 1165 consensus_queue.cc:260] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.545524 1165 ts_tablet_manager.cc:1428] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:57.545790 1165 tablet_bootstrap.cc:492] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.546391 757 tablet_service.cc:1467] Processing CreateTablet for tablet af77e079fe5642d790707406c5588420 (DEFAULT_TABLE table=default.Table_6f7d71e007e84fafa7d6634d3ee5e122 [id=59389c4ca01a45f08d705864669da116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.546696 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af77e079fe5642d790707406c5588420. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:57.558725 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bf72b6386524a4d8a6a883269abc791 consensus state has no leader
I20250109 20:52:57.559484 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 158486f4929e419cbba9d6132c8136ab
W20250109 20:52:57.560077 1922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250109 20:52:57.560415 1922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250109 20:52:57.560537 1922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250109 20:52:57.561491 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 158486f4929e419cbba9d6132c8136ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.562037 1158 tablet_bootstrap.cc:492] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.562243 1158 ts_tablet_manager.cc:1397] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.006s
I20250109 20:52:57.562702 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 158486f4929e419cbba9d6132c8136ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.562726 1176 tablet_bootstrap.cc:492] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.562842 1176 ts_tablet_manager.cc:1397] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
W20250109 20:52:57.563308 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 158486f4929e419cbba9d6132c8136ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 158486f4929e419cbba9d6132c8136ab already in progress: creating tablet
I20250109 20:52:57.563417 1176 raft_consensus.cc:357] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.563374 1158 raft_consensus.cc:357] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.563612 1176 raft_consensus.cc:383] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.563659 1176 raft_consensus.cc:738] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.563637 1158 raft_consensus.cc:383] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.563714 1158 raft_consensus.cc:738] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.563830 1176 consensus_queue.cc:260] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.564301 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 158486f4929e419cbba9d6132c8136ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.564129 1158 consensus_queue.cc:260] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.565145 1158 ts_tablet_manager.cc:1428] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:57.566017 1158 tablet_bootstrap.cc:492] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:57.567209 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 158486f4929e419cbba9d6132c8136ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 158486f4929e419cbba9d6132c8136ab already in progress: creating tablet
I20250109 20:52:57.567874 1176 ts_tablet_manager.cc:1428] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250109 20:52:57.568301 1176 tablet_bootstrap.cc:492] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.572348 1176 tablet_bootstrap.cc:654] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:57.574486 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 158486f4929e419cbba9d6132c8136ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 158486f4929e419cbba9d6132c8136ab already in progress: creating tablet
W20250109 20:52:57.579977 1922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250109 20:52:57.580147 1922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250109 20:52:57.580228 1922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250109 20:52:57.580317 1922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250109 20:52:57.585976 1158 tablet_bootstrap.cc:654] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.594133 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af77e079fe5642d790707406c5588420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.595810 1165 tablet_bootstrap.cc:654] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.596132 1922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=46069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1a98623244882434e9adefa57859e6fff575c2f5
build type FASTDEBUG
built by None at 09 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4099
ASAN enabled
I20250109 20:52:57.599530 1034 tablet_service.cc:1467] Processing CreateTablet for tablet af77e079fe5642d790707406c5588420 (DEFAULT_TABLE table=default.Table_6f7d71e007e84fafa7d6634d3ee5e122 [id=59389c4ca01a45f08d705864669da116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.599794 1922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250109 20:52:57.599943 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af77e079fe5642d790707406c5588420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.601883 1922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250109 20:52:57.604219 889 tablet_service.cc:1467] Processing CreateTablet for tablet 76e688003f5b4a28adfcba0a4ce4b9da (DEFAULT_TABLE table=default.Table_a82e1240e6aa4513b7008ddeb3dff288 [id=a7df3db6006448eeb77618b0f84f0613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.604655 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e688003f5b4a28adfcba0a4ce4b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.609975 1165 tablet_bootstrap.cc:492] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.610167 1165 ts_tablet_manager.cc:1397] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.009s
I20250109 20:52:57.611037 1165 raft_consensus.cc:357] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.611335 1165 raft_consensus.cc:383] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.611435 1165 raft_consensus.cc:738] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.611739 1165 consensus_queue.cc:260] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.611985 2062 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:57.614658 1158 tablet_bootstrap.cc:492] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.614784 1158 ts_tablet_manager.cc:1397] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.007s
I20250109 20:52:57.615460 1158 raft_consensus.cc:357] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.615615 1158 raft_consensus.cc:383] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.615654 1158 raft_consensus.cc:738] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.615880 1158 consensus_queue.cc:260] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.616806 1158 ts_tablet_manager.cc:1428] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:57.617416 1158 tablet_bootstrap.cc:492] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:57.617575 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6dba2d7cf654d449a1c9ff85ac1556f consensus state has no leader
I20250109 20:52:57.619426 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c3c42837503b4d97a261c44ed755ba17" } new_table_name: "default.Table_159068abc4d842c693e5ae2323d1147d" modify_external_catalogs: true
I20250109 20:52:57.620028 1158 tablet_bootstrap.cc:654] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.620934 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_07a5579463ed4880b5f06a89dfb2c469"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.621389 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07a5579463ed4880b5f06a89dfb2c469 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.622480 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cf82774c3759452ba2e9bde7ddc84311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.622895 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf82774c3759452ba2e9bde7ddc84311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.625000 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ea166542339a4980aea4afc8f128edd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.625397 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea166542339a4980aea4afc8f128edd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.627326 446 master_service.cc:946] ReplaceTablet: received request to replace tablet af8c0bf27e994e6aa4cbedca16febe12 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:57.627944 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_63fb1754d95144a3877b430bab3337b6" } modify_external_catalogs: true
I20250109 20:52:57.628085 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_63fb1754d95144a3877b430bab3337b6" } modify_external_catalogs: true
I20250109 20:52:57.631248 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_55c51e87298e485d81fb45c134a1c858" } new_table_name: "default.Table_e01393f526ac40c79f01d50fc724ef49" modify_external_catalogs: true
I20250109 20:52:57.635069 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.635317 1166 tablet_replica.cc:331] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.635474 1166 raft_consensus.cc:2238] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.635604 1166 raft_consensus.cc:2267] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.636158 1166 ts_tablet_manager.cc:1905] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.643375 1166 ts_tablet_manager.cc:1918] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.643810 1166 log.cc:1198] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/0b0450d88e3e4ccaae4e5157ce5c80c1
I20250109 20:52:57.644172 1166 ts_tablet_manager.cc:1939] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.645612 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 (table default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]) successfully deleted
I20250109 20:52:57.649065 1984 raft_consensus.cc:491] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.649197 1984 raft_consensus.cc:513] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.649912 1984 leader_election.cc:290] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:57.650965 713 leader_election.cc:343] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 72a455033b674e5a94fc37d0275eb313
I20250109 20:52:57.653659 758 tablet_service.cc:1467] Processing CreateTablet for tablet 76e688003f5b4a28adfcba0a4ce4b9da (DEFAULT_TABLE table=default.Table_a82e1240e6aa4513b7008ddeb3dff288 [id=a7df3db6006448eeb77618b0f84f0613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.654028 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e688003f5b4a28adfcba0a4ce4b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.613646 1165 ts_tablet_manager.cc:1428] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:57.637959 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b0450d88e3e4ccaae4e5157ce5c80c1) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.661919 1160 tablet_replica.cc:331] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:57.661979 1165 tablet_bootstrap.cc:492] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.662070 1160 raft_consensus.cc:2238] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.662195 1160 raft_consensus.cc:2267] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.660277 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a455033b674e5a94fc37d0275eb313" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:57.663489 1165 tablet_bootstrap.cc:654] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:57.663631 712 leader_election.cc:343] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.663734 712 leader_election.cc:304] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:57.664109 1984 raft_consensus.cc:2743] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:57.665997 889 tablet_service.cc:1467] Processing CreateTablet for tablet f9145358b66d489788ec5cb9d4cd68e5 (DEFAULT_TABLE table=default.Table_df479c8972204ae1a07040b5e2dea99c [id=1ad5d370998145debadb17f254fdd749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.666399 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 76e688003f5b4a28adfcba0a4ce4b9da (DEFAULT_TABLE table=default.Table_a82e1240e6aa4513b7008ddeb3dff288 [id=a7df3db6006448eeb77618b0f84f0613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.666491 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9145358b66d489788ec5cb9d4cd68e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.666839 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e688003f5b4a28adfcba0a4ce4b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.674499 1160 ts_tablet_manager.cc:1905] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.675256 1165 tablet_bootstrap.cc:492] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.675453 1165 ts_tablet_manager.cc:1397] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20250109 20:52:57.676707 1165 raft_consensus.cc:357] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.676961 1165 raft_consensus.cc:383] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.677048 1165 raft_consensus.cc:738] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.677290 1165 consensus_queue.cc:260] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.678438 1984 raft_consensus.cc:491] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.678555 1984 raft_consensus.cc:513] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.679095 1984 leader_election.cc:290] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:57.679811 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8c0bf27e994e6aa4cbedca16febe12" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:57.679992 1165 ts_tablet_manager.cc:1428] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250109 20:52:57.679996 1051 raft_consensus.cc:2463] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:57.680336 713 leader_election.cc:304] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:57.680563 1984 raft_consensus.cc:2798] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:57.680652 1984 raft_consensus.cc:491] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:57.680864 1984 raft_consensus.cc:3054] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:57.681241 1165 tablet_bootstrap.cc:492] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.682623 1165 tablet_bootstrap.cc:654] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.683373 1984 raft_consensus.cc:513] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.684813 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8c0bf27e994e6aa4cbedca16febe12" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:57.685767 712 leader_election.cc:343] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.686332 1984 leader_election.cc:290] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:57.686782 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8c0bf27e994e6aa4cbedca16febe12" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:57.687037 1051 raft_consensus.cc:3054] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:57.689406 1165 tablet_bootstrap.cc:492] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.689565 1165 ts_tablet_manager.cc:1397] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20250109 20:52:57.690100 1051 raft_consensus.cc:2463] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:57.686589 1160 ts_tablet_manager.cc:1918] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.690410 1160 log.cc:1198] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/7d7539cf9ece4befaae6c521d471ea57
I20250109 20:52:57.690492 713 leader_election.cc:304] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:57.690455 1165 raft_consensus.cc:357] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.690863 1165 raft_consensus.cc:383] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.690860 1984 raft_consensus.cc:2798] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:57.690969 1984 raft_consensus.cc:695] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:57.691087 1165 raft_consensus.cc:738] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.691377 1165 consensus_queue.cc:260] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.690774 1160 ts_tablet_manager.cc:1939] T 7d7539cf9ece4befaae6c521d471ea57 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:57.691565 1984 consensus_queue.cc:237] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.692466 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 7d7539cf9ece4befaae6c521d471ea57 (table default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]) successfully deleted
I20250109 20:52:57.693431 1165 ts_tablet_manager.cc:1428] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:57.694942 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8c0bf27e994e6aa4cbedca16febe12" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:57.695065 1165 tablet_bootstrap.cc:492] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:52:57.695392 712 leader_election.cc:343] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.699204 1165 tablet_bootstrap.cc:654] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.700062 753 tablet_service.cc:1467] Processing CreateTablet for tablet f9145358b66d489788ec5cb9d4cd68e5 (DEFAULT_TABLE table=default.Table_df479c8972204ae1a07040b5e2dea99c [id=1ad5d370998145debadb17f254fdd749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.700438 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9145358b66d489788ec5cb9d4cd68e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.701259 1034 tablet_service.cc:1467] Processing CreateTablet for tablet f9145358b66d489788ec5cb9d4cd68e5 (DEFAULT_TABLE table=default.Table_df479c8972204ae1a07040b5e2dea99c [id=1ad5d370998145debadb17f254fdd749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.701582 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9145358b66d489788ec5cb9d4cd68e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.701947 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 111f5e9afc244ad2bd04df715d85bbe9 (DEFAULT_TABLE table=default.Table_735a572f26234eb29120c917e17347c6 [id=91107d468efc4b808ad237f2f6955c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.702237 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111f5e9afc244ad2bd04df715d85bbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.702255 1982 consensus_queue.cc:1035] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:57.702710 758 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf72b6386524a4d8a6a883269abc791 (DEFAULT_TABLE table=default.Table_3e2af93c0f48485e93d72eb35f08e8a3 [id=faa37a6e963e466ea5535a90d4c16455]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.702992 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf72b6386524a4d8a6a883269abc791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.706562 897 tablet_service.cc:1467] Processing CreateTablet for tablet 111f5e9afc244ad2bd04df715d85bbe9 (DEFAULT_TABLE table=default.Table_735a572f26234eb29120c917e17347c6 [id=91107d468efc4b808ad237f2f6955c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.706853 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111f5e9afc244ad2bd04df715d85bbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.706944 757 tablet_service.cc:1467] Processing CreateTablet for tablet 111f5e9afc244ad2bd04df715d85bbe9 (DEFAULT_TABLE table=default.Table_735a572f26234eb29120c917e17347c6 [id=91107d468efc4b808ad237f2f6955c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.707304 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf72b6386524a4d8a6a883269abc791 (DEFAULT_TABLE table=default.Table_3e2af93c0f48485e93d72eb35f08e8a3 [id=faa37a6e963e466ea5535a90d4c16455]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.707909 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf72b6386524a4d8a6a883269abc791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.709257 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111f5e9afc244ad2bd04df715d85bbe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.720533 1176 tablet_bootstrap.cc:492] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.721120 1176 ts_tablet_manager.cc:1397] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.153s user 0.014s sys 0.006s
I20250109 20:52:57.722263 1176 raft_consensus.cc:357] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:57.722960 2072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:57.723577 1176 raft_consensus.cc:383] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.723778 1176 raft_consensus.cc:738] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:52:57.724040 2079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:57.724085 1176 consensus_queue.cc:260] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.725663 1176 ts_tablet_manager.cc:1428] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:57.726300 1176 tablet_bootstrap.cc:492] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.728489 1176 tablet_bootstrap.cc:654] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.731417 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_47acbd3655b34a70bdefe7ab105d58a1" } modify_external_catalogs: true
I20250109 20:52:57.731715 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_47acbd3655b34a70bdefe7ab105d58a1" } modify_external_catalogs: true
I20250109 20:52:57.733052 1922 server_base.cc:1034] running on GCE node
I20250109 20:52:57.730054 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.730544 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.730907 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 with delete_type TABLET_DATA_DELETED (replaced by tablet beca5f1ed9d3434fb7de90cd74a6f09b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.734726 1166 tablet_replica.cc:331] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:57.734767 2081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250109 20:52:57.734958 1166 raft_consensus.cc:2238] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.735095 1166 raft_consensus.cc:2267] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.735967 1166 ts_tablet_manager.cc:1905] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.736042 1922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250109 20:52:57.738133 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025cdef634d4626a33f01c362a1388a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.738385 1157 tablet_replica.cc:331] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:57.738538 1157 raft_consensus.cc:2238] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.738521 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:57.738660 1157 raft_consensus.cc:2267] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.738669 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 2608f48f5950466e9ab2d12ee87503cb (table default.Table_7336cda1ce164722949bdefb9f26e7cd [id=17d12fac83d6463fb91fab6729a08139]). Expected version 2 got 0
I20250109 20:52:57.739187 1157 ts_tablet_manager.cc:1905] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.741148 454 master_service.cc:946] ReplaceTablet: received request to replace tablet c00aaf81b34445abb7502700f01d4f3f from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:57.741820 1166 ts_tablet_manager.cc:1918] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.742028 1166 log.cc:1198] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/219a4b508050490cb3f1ee418e0edbf8
I20250109 20:52:57.742486 1166 ts_tablet_manager.cc:1939] T 219a4b508050490cb3f1ee418e0edbf8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.743291 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc with delete_type TABLET_DATA_DELETED (replaced by tablet 829bec4f66c24180afd5271057a7d0e8) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.743652 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 219a4b508050490cb3f1ee418e0edbf8 (table default.Table_8d504623c1894cb8843ead0ee687ea13 [id=5c7e66cff6b14f9a81029cea36224460]) successfully deleted
I20250109 20:52:57.743909 449 master_service.cc:946] ReplaceTablet: received request to replace tablet a1ed00f7954143c9ba9bf65d8a8a68d2 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:57.744269 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e4bfa188b545349f358adfd0bc714b with delete_type TABLET_DATA_DELETED (replaced by tablet a1ed00f7954143c9ba9bf65d8a8a68d2) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.744351 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49ecdce4b5a94e4a92d868acdaefb783" } modify_external_catalogs: true
I20250109 20:52:57.744446 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1630e4438e4337ba026b41f630a480 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.744493 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_49ecdce4b5a94e4a92d868acdaefb783" } modify_external_catalogs: true
I20250109 20:52:57.744540 1166 tablet_replica.cc:331] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.744446 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a455033b674e5a94fc37d0275eb313 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.744735 1166 raft_consensus.cc:2238] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.745303 1157 ts_tablet_manager.cc:1918] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.745339 1166 raft_consensus.cc:2267] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.745489 1157 log.cc:1198] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4025cdef634d4626a33f01c362a1388a
I20250109 20:52:57.745836 1157 ts_tablet_manager.cc:1939] T 4025cdef634d4626a33f01c362a1388a P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:57.746896 1166 ts_tablet_manager.cc:1905] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.747489 918 raft_consensus.cc:3054] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:57.747685 1922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250109 20:52:57.749974 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4025cdef634d4626a33f01c362a1388a (table default.Table_d52320f2397e4526ae456263f71bcda2 [id=0b96e2cb7e234f498b8ef5e9a09e9a67]) successfully deleted
I20250109 20:52:57.750288 1157 tablet_replica.cc:331] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:57.750492 1157 raft_consensus.cc:2238] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.750664 1157 raft_consensus.cc:2267] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.751286 1157 ts_tablet_manager.cc:1905] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.752480 1166 ts_tablet_manager.cc:1918] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.752581 1166 log.cc:1198] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/8890a6a93e3d48dc96955f1b51e78603
W20250109 20:52:57.753377 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f3c276fab6a4d068d64ea5d23c9a595 consensus state has no leader
I20250109 20:52:57.756613 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7539cf9ece4befaae6c521d471ea57 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b0450d88e3e4ccaae4e5157ce5c80c1) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.757076 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8e7081cac440389bec5dfac2391beb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:57.757349 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7d7539cf9ece4befaae6c521d471ea57 because the tablet was not found. No further retry: Not found: Tablet not found: 7d7539cf9ece4befaae6c521d471ea57
I20250109 20:52:57.758862 1160 tablet_replica.cc:331] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:57.759006 1160 raft_consensus.cc:2238] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.759126 1160 raft_consensus.cc:2267] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:57.759754 712 consensus_peers.cc:487] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:57.760193 1160 ts_tablet_manager.cc:1905] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.762778 1922 hybrid_clock.cc:648] HybridClock initialized: now 1736455977762649 us; error 107 us; skew 500 ppm
I20250109 20:52:57.763185 1922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250109 20:52:57.765463 1166 ts_tablet_manager.cc:1939] T 8890a6a93e3d48dc96955f1b51e78603 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.767194 1176 tablet_bootstrap.cc:492] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.767295 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 8890a6a93e3d48dc96955f1b51e78603 (table default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]) successfully deleted
W20250109 20:52:57.767554 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 because the tablet was not found. No further retry: Not found: Tablet not found: 0b0450d88e3e4ccaae4e5157ce5c80c1
I20250109 20:52:57.769078 1160 ts_tablet_manager.cc:1918] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.769994 1160 log.cc:1198] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4d8e7081cac440389bec5dfac2391beb
I20250109 20:52:57.770344 1176 ts_tablet_manager.cc:1397] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.001s
I20250109 20:52:57.770428 1160 ts_tablet_manager.cc:1939] T 4d8e7081cac440389bec5dfac2391beb P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:57.775420 914 raft_consensus.cc:3054] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:57.776979 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf72b6386524a4d8a6a883269abc791 (DEFAULT_TABLE table=default.Table_3e2af93c0f48485e93d72eb35f08e8a3 [id=faa37a6e963e466ea5535a90d4c16455]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.777433 897 tablet_service.cc:1467] Processing CreateTablet for tablet 28135d08f8364311ba26a739ea0f910a (DEFAULT_TABLE table=default.Table_c58a40d4195d443d82138e5eedbb7b93 [id=dba369946b744363a5eb9e9eaf128a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.777552 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf72b6386524a4d8a6a883269abc791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.777519 1176 raft_consensus.cc:357] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.777680 1176 raft_consensus.cc:383] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.777719 1176 raft_consensus.cc:738] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.777745 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28135d08f8364311ba26a739ea0f910a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.777868 1176 consensus_queue.cc:260] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.778090 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 28135d08f8364311ba26a739ea0f910a (DEFAULT_TABLE table=default.Table_c58a40d4195d443d82138e5eedbb7b93 [id=dba369946b744363a5eb9e9eaf128a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.778311 1176 ts_tablet_manager.cc:1428] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250109 20:52:57.778509 1176 tablet_bootstrap.cc:492] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.778733 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28135d08f8364311ba26a739ea0f910a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.780884 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 27595dcecf06474c8f699da322581a07 (DEFAULT_TABLE table=default.Table_23ea215227ba4c4ea8414587c0e2d085 [id=38c8adf2be454c9cae3bb0fb11e51e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.780917 886 tablet_service.cc:1467] Processing CreateTablet for tablet 27595dcecf06474c8f699da322581a07 (DEFAULT_TABLE table=default.Table_23ea215227ba4c4ea8414587c0e2d085 [id=38c8adf2be454c9cae3bb0fb11e51e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.781162 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27595dcecf06474c8f699da322581a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.781221 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27595dcecf06474c8f699da322581a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.781771 885 tablet_service.cc:1467] Processing CreateTablet for tablet 643e337667d34534b88715704ab88e82 (DEFAULT_TABLE table=default.Table_1a922ef3992c4ec088cfdf98b527dcf2 [id=a0850fba0c514390a228da8116f415e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.782042 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643e337667d34534b88715704ab88e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.783934 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 643e337667d34534b88715704ab88e82 (DEFAULT_TABLE table=default.Table_1a922ef3992c4ec088cfdf98b527dcf2 [id=a0850fba0c514390a228da8116f415e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.784179 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643e337667d34534b88715704ab88e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.785826 750 tablet_service.cc:1467] Processing CreateTablet for tablet 27595dcecf06474c8f699da322581a07 (DEFAULT_TABLE table=default.Table_23ea215227ba4c4ea8414587c0e2d085 [id=38c8adf2be454c9cae3bb0fb11e51e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.786005 758 tablet_service.cc:1467] Processing CreateTablet for tablet 28135d08f8364311ba26a739ea0f910a (DEFAULT_TABLE table=default.Table_c58a40d4195d443d82138e5eedbb7b93 [id=dba369946b744363a5eb9e9eaf128a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.786196 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27595dcecf06474c8f699da322581a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.786540 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28135d08f8364311ba26a739ea0f910a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.787751 1176 tablet_bootstrap.cc:654] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:57.787828 712 consensus_peers.cc:487] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:57.788976 755 tablet_service.cc:1467] Processing CreateTablet for tablet 643e337667d34534b88715704ab88e82 (DEFAULT_TABLE table=default.Table_1a922ef3992c4ec088cfdf98b527dcf2 [id=a0850fba0c514390a228da8116f415e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.789307 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643e337667d34534b88715704ab88e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.800767 1176 tablet_bootstrap.cc:492] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.801029 1176 ts_tablet_manager.cc:1397] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20250109 20:52:57.801882 1176 raft_consensus.cc:357] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.802270 1176 raft_consensus.cc:383] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.802350 1176 raft_consensus.cc:738] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.802665 1176 consensus_queue.cc:260] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.803567 1176 ts_tablet_manager.cc:1428] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:57.803831 1176 tablet_bootstrap.cc:492] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.804225 1157 ts_tablet_manager.cc:1918] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.805573 1176 tablet_bootstrap.cc:654] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.805799 1157 log.cc:1198] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/705a7c04f3f948e1a5e486d5ccd48ccc
I20250109 20:52:57.806540 1922 webserver.cc:458] Webserver started at http://127.0.104.60:46069/ using document root <none> and password file <none>
I20250109 20:52:57.806674 1157 ts_tablet_manager.cc:1939] T 705a7c04f3f948e1a5e486d5ccd48ccc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:57.808015 1922 fs_manager.cc:362] Metadata directory not provided
I20250109 20:52:57.808264 1922 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250109 20:52:57.808605 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 705a7c04f3f948e1a5e486d5ccd48ccc (table default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]) successfully deleted
I20250109 20:52:57.810659 1157 tablet_replica.cc:331] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:57.811242 1157 raft_consensus.cc:2238] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.811836 1157 raft_consensus.cc:2267] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.812655 1157 ts_tablet_manager.cc:1905] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.818933 1157 ts_tablet_manager.cc:1918] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.822877 1922 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.004s sys 0.000s
I20250109 20:52:57.826040 1176 tablet_bootstrap.cc:492] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:57.826324 1176 ts_tablet_manager.cc:1397] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.003s
I20250109 20:52:57.826993 1176 raft_consensus.cc:357] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.827319 1176 raft_consensus.cc:383] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.827392 1176 raft_consensus.cc:738] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.827615 1176 consensus_queue.cc:260] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.828156 1176 ts_tablet_manager.cc:1428] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:52:57.828409 1176 tablet_bootstrap.cc:492] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:57.829985 1176 tablet_bootstrap.cc:654] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.832296 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4d8e7081cac440389bec5dfac2391beb (table default.Table_6c20eaf8f2c14521b3a0a0125d2f5cb3 [id=e1afc4c2550f49f4b3c7b17ea2f09f60]) successfully deleted
I20250109 20:52:57.838713 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 012ce4d7dbe14a0795497fd3be2f9feb (DEFAULT_TABLE table=default.Table_55c51e87298e485d81fb45c134a1c858 [id=2db0a92c98b845b0afbea35b957d2cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.839054 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012ce4d7dbe14a0795497fd3be2f9feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.840123 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d2c8a6555a8e4228a9a80b15c6880c84 (DEFAULT_TABLE table=default.Table_b1e5a40512794ab9ad99471666d8f7b7 [id=a3784b75df1c44e4b72d386b8e769db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.840440 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c8a6555a8e4228a9a80b15c6880c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.843673 887 tablet_service.cc:1467] Processing CreateTablet for tablet b87ab6633b774af5a6d258a7efd07472 (DEFAULT_TABLE table=default.Table_46c18c9d597b4fb1972f05a7c835e2a5 [id=41c9641de00046d885f7eb0928a26976]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.844009 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87ab6633b774af5a6d258a7efd07472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.844437 1020 tablet_service.cc:1467] Processing CreateTablet for tablet b87ab6633b774af5a6d258a7efd07472 (DEFAULT_TABLE table=default.Table_46c18c9d597b4fb1972f05a7c835e2a5 [id=41c9641de00046d885f7eb0928a26976]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.844626 1026 tablet_service.cc:1467] Processing CreateTablet for tablet d0e0f4e3a82a437ebbb16426fffa5e2f (DEFAULT_TABLE table=default.Table_0192f16726024e458eb0eed5b1b5dfd2 [id=51a58f4a2e5e4fb3890abd11e22e8110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.844717 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87ab6633b774af5a6d258a7efd07472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.849251 897 tablet_service.cc:1467] Processing CreateTablet for tablet d2c8a6555a8e4228a9a80b15c6880c84 (DEFAULT_TABLE table=default.Table_b1e5a40512794ab9ad99471666d8f7b7 [id=a3784b75df1c44e4b72d386b8e769db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.849393 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e0f4e3a82a437ebbb16426fffa5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.845499 1016 tablet_service.cc:1467] Processing CreateTablet for tablet af6d061fd1e34117bd8ba609c0638d48 (DEFAULT_TABLE table=default.Table_44d9cfca7ff645bb8aad32946b74f286 [id=a802ce427a7d45b689bcd06b0aee1e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.849553 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c8a6555a8e4228a9a80b15c6880c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.849756 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af6d061fd1e34117bd8ba609c0638d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.849152 1029 tablet_service.cc:1467] Processing CreateTablet for tablet fb3e620799a943c6850e20a3d6d2f6bb (DEFAULT_TABLE table=default.Table_8b869f8172ed4fcf815c58fc296752c6 [id=838faaf3c00c4ec582e8eeb78a7db528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.852334 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3e620799a943c6850e20a3d6d2f6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.853076 885 tablet_service.cc:1467] Processing CreateTablet for tablet d0e0f4e3a82a437ebbb16426fffa5e2f (DEFAULT_TABLE table=default.Table_0192f16726024e458eb0eed5b1b5dfd2 [id=51a58f4a2e5e4fb3890abd11e22e8110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.853335 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e0f4e3a82a437ebbb16426fffa5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.858650 886 tablet_service.cc:1467] Processing CreateTablet for tablet 012ce4d7dbe14a0795497fd3be2f9feb (DEFAULT_TABLE table=default.Table_55c51e87298e485d81fb45c134a1c858 [id=2db0a92c98b845b0afbea35b957d2cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.858978 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012ce4d7dbe14a0795497fd3be2f9feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.860488 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cca642fb9c3e4a55adfb6ad8386a67cd
I20250109 20:52:57.862082 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet cca642fb9c3e4a55adfb6ad8386a67cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:57.863683 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cca642fb9c3e4a55adfb6ad8386a67cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cca642fb9c3e4a55adfb6ad8386a67cd already in progress: creating tablet
I20250109 20:52:57.864571 896 tablet_service.cc:1514] Processing DeleteTablet for tablet cca642fb9c3e4a55adfb6ad8386a67cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:57.865270 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cca642fb9c3e4a55adfb6ad8386a67cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cca642fb9c3e4a55adfb6ad8386a67cd already in progress: creating tablet
I20250109 20:52:57.873380 2091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250109 20:52:57.875669 1922 fs_manager.cc:730] Time spent opening block manager: real 0.041s user 0.001s sys 0.000s
I20250109 20:52:57.876006 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e4aa837977f646d1962fa4a45ea438b2" } new_table_name: "default.Table_bb47b9b8617f4706aeb895a16ded22ee" modify_external_catalogs: true
I20250109 20:52:57.876235 889 tablet_service.cc:1467] Processing CreateTablet for tablet fb3e620799a943c6850e20a3d6d2f6bb (DEFAULT_TABLE table=default.Table_8b869f8172ed4fcf815c58fc296752c6 [id=838faaf3c00c4ec582e8eeb78a7db528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.876570 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3e620799a943c6850e20a3d6d2f6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.877398 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4887e41bfb6f4a4aa53cb437224c8e1c" } modify_external_catalogs: true
I20250109 20:52:57.877557 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4887e41bfb6f4a4aa53cb437224c8e1c" } modify_external_catalogs: true
I20250109 20:52:57.879727 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]). Expected version 3 got 0
I20250109 20:52:57.876142 1922 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
uuid: "36a29ab2002d4eb389cd11778988e1e9"
format_stamp: "Formatted at 2025-01-09 20:52:40 on dist-test-slave-612g"
I20250109 20:52:57.879916 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
I20250109 20:52:57.879927 1922 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250109 20:52:57.880609 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 5b107b4c26a94898b744654195bfe567 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:57.882952 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c0d1dadec3324eee8cc42a24ab27f3ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.883481 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0d1dadec3324eee8cc42a24ab27f3ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.885519 893 tablet_service.cc:1467] Processing CreateTablet for tablet af6d061fd1e34117bd8ba609c0638d48 (DEFAULT_TABLE table=default.Table_44d9cfca7ff645bb8aad32946b74f286 [id=a802ce427a7d45b689bcd06b0aee1e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.885895 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af6d061fd1e34117bd8ba609c0638d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.891362 1165 tablet_bootstrap.cc:492] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:57.891551 1165 ts_tablet_manager.cc:1397] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.197s user 0.012s sys 0.008s
I20250109 20:52:57.892403 1165 raft_consensus.cc:357] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.892617 1165 raft_consensus.cc:383] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:57.892732 1165 raft_consensus.cc:738] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:57.892982 1165 consensus_queue.cc:260] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.893714 1165 ts_tablet_manager.cc:1428] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250109 20:52:57.899614 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 012ce4d7dbe14a0795497fd3be2f9feb consensus state has no leader
I20250109 20:52:57.900409 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3008c672bc544000b24b75ae366d9fb9
I20250109 20:52:57.901784 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 3008c672bc544000b24b75ae366d9fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:57.902501 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3008c672bc544000b24b75ae366d9fb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3008c672bc544000b24b75ae366d9fb9 already in progress: creating tablet
I20250109 20:52:57.902743 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet af8c0bf27e994e6aa4cbedca16febe12
I20250109 20:52:57.903334 446 catalog_manager.cc:6596] ReplaceTablet: tablet af8c0bf27e994e6aa4cbedca16febe12 deleted and replaced by tablet 6cb88771d2b148bdae6e427c1b25b9a9
I20250109 20:52:57.903930 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet af8c0bf27e994e6aa4cbedca16febe12 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cb88771d2b148bdae6e427c1b25b9a9) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.904168 1166 tablet_replica.cc:331] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.904349 1166 raft_consensus.cc:2238] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.904464 1166 raft_consensus.cc:2267] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.904975 1166 ts_tablet_manager.cc:1905] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.911190 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
I20250109 20:52:57.911731 1166 ts_tablet_manager.cc:1918] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.911820 1166 log.cc:1198] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/af8c0bf27e994e6aa4cbedca16febe12
I20250109 20:52:57.913260 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cb6e0dd9d8ff4cde977360e8da20fd10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:57.913733 1165 tablet_bootstrap.cc:492] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:57.914709 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.914927 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 with delete_type TABLET_DATA_DELETED (replaced by tablet b6b23623677b4e088c14af925b4e07d4) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:57.915001 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb6e0dd9d8ff4cde977360e8da20fd10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.915103 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 with delete_type TABLET_DATA_DELETED (replaced by tablet d79437d800f44c5a8222eea7dc2cd4e3) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.915378 1165 tablet_bootstrap.cc:654] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:57.915515 1166 ts_tablet_manager.cc:1939] T af8c0bf27e994e6aa4cbedca16febe12 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.914469 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0ad7f647b70b4047addd805f70288382" } new_table_name: "default.Table_aef4bd2a519c4d27964b01f6449b516b" modify_external_catalogs: true
I20250109 20:52:57.902789 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 3008c672bc544000b24b75ae366d9fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.907477 885 tablet_service.cc:1514] Processing DeleteTablet for tablet af8c0bf27e994e6aa4cbedca16febe12 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cb88771d2b148bdae6e427c1b25b9a9) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.916450 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet af8c0bf27e994e6aa4cbedca16febe12 (table default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]) successfully deleted
I20250109 20:52:57.916826 1166 tablet_replica.cc:331] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:57.916991 1166 raft_consensus.cc:2238] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.917132 1166 raft_consensus.cc:2267] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.917764 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_113e0f4bda2b441986056ee8e3616d94" } modify_external_catalogs: true
I20250109 20:52:57.917927 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_113e0f4bda2b441986056ee8e3616d94" } modify_external_catalogs: true
I20250109 20:52:57.918161 1166 ts_tablet_manager.cc:1905] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:57.919875 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3008c672bc544000b24b75ae366d9fb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3008c672bc544000b24b75ae366d9fb9 already in progress: creating tablet
W20250109 20:52:57.920090 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet af8c0bf27e994e6aa4cbedca16febe12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af8c0bf27e994e6aa4cbedca16febe12 already in progress: creating tablet
I20250109 20:52:57.920776 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_23f5de34735e490a84de4f459a0258f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.921222 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23f5de34735e490a84de4f459a0258f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.921764 1960 raft_consensus.cc:491] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.921898 1960 raft_consensus.cc:513] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.922573 1960 leader_election.cc:290] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:52:57.923431 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda300a7e58a44339c3bdb2e62a9f384" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:57.923862 989 leader_election.cc:343] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.925421 885 tablet_service.cc:1467] Processing CreateTablet for tablet eaea4f239cce41a997496a1c94810ac9 (DEFAULT_TABLE table=default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.925866 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaea4f239cce41a997496a1c94810ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.926056 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c00aaf81b34445abb7502700f01d4f3f
I20250109 20:52:57.926537 454 catalog_manager.cc:6596] ReplaceTablet: tablet c00aaf81b34445abb7502700f01d4f3f deleted and replaced by tablet 795f0f29f0744f4e9dd2fcf1762ab978
I20250109 20:52:57.926898 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet c00aaf81b34445abb7502700f01d4f3f with delete_type TABLET_DATA_DELETED (replaced by tablet 795f0f29f0744f4e9dd2fcf1762ab978) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.928287 1016 tablet_service.cc:1467] Processing CreateTablet for tablet eaea4f239cce41a997496a1c94810ac9 (DEFAULT_TABLE table=default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.928757 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaea4f239cce41a997496a1c94810ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.928845 886 tablet_service.cc:1514] Processing DeleteTablet for tablet c00aaf81b34445abb7502700f01d4f3f with delete_type TABLET_DATA_DELETED (replaced by tablet 795f0f29f0744f4e9dd2fcf1762ab978) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.929236 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:57.929415 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:57.930624 1160 tablet_replica.cc:331] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:57.930775 1160 raft_consensus.cc:2238] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:57.931051 1160 raft_consensus.cc:2267] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.931496 1160 ts_tablet_manager.cc:1905] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:57.838465 750 tablet_service.cc:1467] Processing CreateTablet for tablet 012ce4d7dbe14a0795497fd3be2f9feb (DEFAULT_TABLE table=default.Table_55c51e87298e485d81fb45c134a1c858 [id=2db0a92c98b845b0afbea35b957d2cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.933964 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012ce4d7dbe14a0795497fd3be2f9feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.841068 755 tablet_service.cc:1467] Processing CreateTablet for tablet d0e0f4e3a82a437ebbb16426fffa5e2f (DEFAULT_TABLE table=default.Table_0192f16726024e458eb0eed5b1b5dfd2 [id=51a58f4a2e5e4fb3890abd11e22e8110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.936992 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e0f4e3a82a437ebbb16426fffa5e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.937501 1160 ts_tablet_manager.cc:1918] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:57.937577 1160 log.cc:1198] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/c00aaf81b34445abb7502700f01d4f3f
I20250109 20:52:57.937847 1160 ts_tablet_manager.cc:1939] T c00aaf81b34445abb7502700f01d4f3f P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:57.939425 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
I20250109 20:52:57.940202 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_19300a35d595402fbb4d39933c3ba00c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:57.940724 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19300a35d595402fbb4d39933c3ba00c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.940794 1166 ts_tablet_manager.cc:1918] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:57.940912 1166 log.cc:1198] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ffd3a1b1d3e44feda6dbaa8556e377fc
I20250109 20:52:57.941259 1166 ts_tablet_manager.cc:1939] T ffd3a1b1d3e44feda6dbaa8556e377fc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:57.841394 753 tablet_service.cc:1467] Processing CreateTablet for tablet d2c8a6555a8e4228a9a80b15c6880c84 (DEFAULT_TABLE table=default.Table_b1e5a40512794ab9ad99471666d8f7b7 [id=a3784b75df1c44e4b72d386b8e769db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.942832 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c8a6555a8e4228a9a80b15c6880c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.943837 745 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe6bd7bc23a4d8095f025b01dd5933c (DEFAULT_TABLE table=default.Table_b2efe8b52c934acaac976ac5af84531e [id=becef7430b45418aa8d60851bb2551a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.944229 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe6bd7bc23a4d8095f025b01dd5933c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.842329 740 tablet_service.cc:1467] Processing CreateTablet for tablet fb3e620799a943c6850e20a3d6d2f6bb (DEFAULT_TABLE table=default.Table_8b869f8172ed4fcf815c58fc296752c6 [id=838faaf3c00c4ec582e8eeb78a7db528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.843199 747 tablet_service.cc:1467] Processing CreateTablet for tablet b87ab6633b774af5a6d258a7efd07472 (DEFAULT_TABLE table=default.Table_46c18c9d597b4fb1972f05a7c835e2a5 [id=41c9641de00046d885f7eb0928a26976]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.845157 756 tablet_service.cc:1467] Processing CreateTablet for tablet af6d061fd1e34117bd8ba609c0638d48 (DEFAULT_TABLE table=default.Table_44d9cfca7ff645bb8aad32946b74f286 [id=a802ce427a7d45b689bcd06b0aee1e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.849854 739 tablet_service.cc:1467] Processing CreateTablet for tablet eaea4f239cce41a997496a1c94810ac9 (DEFAULT_TABLE table=default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.871161 757 tablet_service.cc:1514] Processing DeleteTablet for tablet cca642fb9c3e4a55adfb6ad8386a67cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.903496 751 tablet_service.cc:1514] Processing DeleteTablet for tablet af8c0bf27e994e6aa4cbedca16febe12 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cb88771d2b148bdae6e427c1b25b9a9) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.908314 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 3008c672bc544000b24b75ae366d9fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.909186 752 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.909941 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 with delete_type TABLET_DATA_DELETED (replaced by tablet e16a1d9a5b374b8dbf725eb32ffadb9e) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.910061 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee with delete_type TABLET_DATA_DELETED (replaced by tablet d9f18f244a344272bba2bc085686b61d) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.923764 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda300a7e58a44339c3bdb2e62a9f384" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:57.926653 746 tablet_service.cc:1514] Processing DeleteTablet for tablet c00aaf81b34445abb7502700f01d4f3f with delete_type TABLET_DATA_DELETED (replaced by tablet 795f0f29f0744f4e9dd2fcf1762ab978) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:57.949985 990 leader_election.cc:343] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.950130 990 leader_election.cc:304] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:57.950559 1960 raft_consensus.cc:2743] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:57.950767 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb3e620799a943c6850e20a3d6d2f6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.955528 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87ab6633b774af5a6d258a7efd07472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.959832 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af6d061fd1e34117bd8ba609c0638d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.961128 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bcd3219734314ae1ad03968fdf9c9cd0
I20250109 20:52:57.962319 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd3219734314ae1ad03968fdf9c9cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.962626 745 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd3219734314ae1ad03968fdf9c9cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:57.962836 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:57.962996 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483 [id=3e073742d9cf476387b0a0949059d933]). Expected version 1 got 0
I20250109 20:52:57.964670 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaea4f239cce41a997496a1c94810ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.965390 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]). Expected version 1 got 0
I20250109 20:52:57.965951 1960 raft_consensus.cc:491] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:57.966099 1960 raft_consensus.cc:513] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:57.966372 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_69d6ced747db4140aee267f44cad8d13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:57.966830 1960 leader_election.cc:290] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:57.966995 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69d6ced747db4140aee267f44cad8d13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.967460 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0538f4d3608a454b9dd21a0fa42068a0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:57.968024 990 leader_election.cc:343] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.970058 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe6bd7bc23a4d8095f025b01dd5933c (DEFAULT_TABLE table=default.Table_b2efe8b52c934acaac976ac5af84531e [id=becef7430b45418aa8d60851bb2551a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.970520 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe6bd7bc23a4d8095f025b01dd5933c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.970947 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_385822fe7219471bacdd1ad2dbe7d891 [id=0430588a51ed49ae86290eaab54f79dc]) successfully deleted
I20250109 20:52:57.971484 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5d462230734767bb010f520c86c5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.971710 1160 tablet_replica.cc:331] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:57.971698 887 tablet_service.cc:1467] Processing CreateTablet for tablet 90bcfd699e094c499df625281e7119b1 (DEFAULT_TABLE table=default.Table_ebdd59dd13424ecf93c997fa5919477d [id=79febb0a1443466aaabc6d31347d6ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.971947 1160 raft_consensus.cc:2238] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:57.972127 1160 raft_consensus.cc:2267] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:57.975337 1160 ts_tablet_manager.cc:1905] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:57.979427 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2420b285905342b3820f9549c8f0c4ad consensus state has no leader
I20250109 20:52:57.979534 747 tablet_service.cc:1467] Processing CreateTablet for tablet 90bcfd699e094c499df625281e7119b1 (DEFAULT_TABLE table=default.Table_ebdd59dd13424ecf93c997fa5919477d [id=79febb0a1443466aaabc6d31347d6ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.980399 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90bcfd699e094c499df625281e7119b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.980984 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c71e6da6e81e4bb3becf5e6834b8a4ff" } new_table_name: "default.Table_1c1f3e1543534694a3ccf10b8bf07284" modify_external_catalogs: true
I20250109 20:52:57.983881 890 tablet_service.cc:1467] Processing CreateTablet for tablet e6dba2d7cf654d449a1c9ff85ac1556f (DEFAULT_TABLE table=default.Table_de7dc1f873f44dcb9e4eb13f73a25657 [id=69a3c4ad65704ac5a50690d19330171a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.984227 740 tablet_service.cc:1467] Processing CreateTablet for tablet e6dba2d7cf654d449a1c9ff85ac1556f (DEFAULT_TABLE table=default.Table_de7dc1f873f44dcb9e4eb13f73a25657 [id=69a3c4ad65704ac5a50690d19330171a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.984314 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e6dba2d7cf654d449a1c9ff85ac1556f (DEFAULT_TABLE table=default.Table_de7dc1f873f44dcb9e4eb13f73a25657 [id=69a3c4ad65704ac5a50690d19330171a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.984778 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6dba2d7cf654d449a1c9ff85ac1556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.984911 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6dba2d7cf654d449a1c9ff85ac1556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.988308 886 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd3219734314ae1ad03968fdf9c9cd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.988535 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 76bd758fa2fa4cebbdf7faf3856f1941
I20250109 20:52:57.989251 450 master_service.cc:946] ReplaceTablet: received request to replace tablet a1ed00f7954143c9ba9bf65d8a8a68d2 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:57.989579 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 76bd758fa2fa4cebbdf7faf3856f1941 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:57.990033 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 76bd758fa2fa4cebbdf7faf3856f1941 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.990643 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0538f4d3608a454b9dd21a0fa42068a0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:57.991370 989 leader_election.cc:343] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:57.991500 989 leader_election.cc:304] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:57.991711 1960 raft_consensus.cc:2743] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:57.991828 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1ce8dba031c24ab2bd13cbaa5d99c9fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:57.992059 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 76bd758fa2fa4cebbdf7faf3856f1941 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:57.992331 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ce8dba031c24ab2bd13cbaa5d99c9fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:57.994453 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 90bcfd699e094c499df625281e7119b1 (DEFAULT_TABLE table=default.Table_ebdd59dd13424ecf93c997fa5919477d [id=79febb0a1443466aaabc6d31347d6ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:57.994983 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90bcfd699e094c499df625281e7119b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:57.996327 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d96a569e6be4e01a26918d115cfbcf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:57.933632 1158 tablet_bootstrap.cc:492] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:57.996879 1158 ts_tablet_manager.cc:1397] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.379s user 0.030s sys 0.026s
I20250109 20:52:57.998507 1158 raft_consensus.cc:357] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.000568 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90bcfd699e094c499df625281e7119b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.000489 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4e532273983a46e8ba2890fb8c8ce149 (DEFAULT_TABLE table=default.Table_12b56043fe0e4f4681a9522f8d62d94c [id=e3859c6cd4d24ae3947836b32ce047a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.000797 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6dba2d7cf654d449a1c9ff85ac1556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.000921 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e532273983a46e8ba2890fb8c8ce149. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.001529 1157 log.cc:1198] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/66e4bfa188b545349f358adfd0bc714b
I20250109 20:52:58.001935 1157 ts_tablet_manager.cc:1939] T 66e4bfa188b545349f358adfd0bc714b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.003036 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 66e4bfa188b545349f358adfd0bc714b (table default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]) successfully deleted
I20250109 20:52:58.003365 1157 tablet_replica.cc:331] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:58.003593 1157 raft_consensus.cc:2238] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.003799 1157 raft_consensus.cc:2267] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.004305 1157 ts_tablet_manager.cc:1905] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.005980 885 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe6bd7bc23a4d8095f025b01dd5933c (DEFAULT_TABLE table=default.Table_b2efe8b52c934acaac976ac5af84531e [id=becef7430b45418aa8d60851bb2551a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.008704 1176 tablet_bootstrap.cc:492] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.008997 1176 ts_tablet_manager.cc:1397] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.181s user 0.012s sys 0.013s
I20250109 20:52:58.009985 1176 raft_consensus.cc:357] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.010278 1176 raft_consensus.cc:383] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.010380 1176 raft_consensus.cc:738] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.010716 1176 consensus_queue.cc:260] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.015066 1176 ts_tablet_manager.cc:1428] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250109 20:52:58.016065 889 tablet_service.cc:1467] Processing CreateTablet for tablet 4e532273983a46e8ba2890fb8c8ce149 (DEFAULT_TABLE table=default.Table_12b56043fe0e4f4681a9522f8d62d94c [id=e3859c6cd4d24ae3947836b32ce047a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.016489 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e532273983a46e8ba2890fb8c8ce149. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.016876 1176 tablet_bootstrap.cc:492] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.018404 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe6bd7bc23a4d8095f025b01dd5933c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.019685 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4e532273983a46e8ba2890fb8c8ce149 (DEFAULT_TABLE table=default.Table_12b56043fe0e4f4681a9522f8d62d94c [id=e3859c6cd4d24ae3947836b32ce047a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.020114 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e532273983a46e8ba2890fb8c8ce149. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.022281 886 tablet_service.cc:1467] Processing CreateTablet for tablet d36e80f0671f404bbd0e2dded93716a3 (DEFAULT_TABLE table=default.Table_486f2f72d50e49cdb7e8a1c022cb439d [id=475d752e44c14cf7bdf3d447356f0c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.022657 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36e80f0671f404bbd0e2dded93716a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.023398 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3a253685e34e80a26f338be66d31bb (DEFAULT_TABLE table=default.Table_17e950d4133f4d2a8eeab7f7af0b10c2 [id=e954185bc21e421f824d516f2cf4285b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.023667 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3a253685e34e80a26f338be66d31bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.024389 739 tablet_service.cc:1467] Processing CreateTablet for tablet c3d9f992800e4a41a7af7e2751214eaa (DEFAULT_TABLE table=default.Table_5cf48a44b13a4059b1e8974875e736e2 [id=7080faa7b3844702859c8a48be032b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.024813 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d9f992800e4a41a7af7e2751214eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.025079 880 tablet_service.cc:1467] Processing CreateTablet for tablet 527375706fef48fda1adeeddf2434cb5 (DEFAULT_TABLE table=default.Table_a4fac4fff616461d9e0b116e98bdd5e8 [id=b82dad04c5514172867f93c52cb39722]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.025388 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527375706fef48fda1adeeddf2434cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.028456 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3a253685e34e80a26f338be66d31bb (DEFAULT_TABLE table=default.Table_17e950d4133f4d2a8eeab7f7af0b10c2 [id=e954185bc21e421f824d516f2cf4285b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.028568 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d36e80f0671f404bbd0e2dded93716a3 (DEFAULT_TABLE table=default.Table_486f2f72d50e49cdb7e8a1c022cb439d [id=475d752e44c14cf7bdf3d447356f0c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.029170 755 tablet_service.cc:1467] Processing CreateTablet for tablet 527375706fef48fda1adeeddf2434cb5 (DEFAULT_TABLE table=default.Table_a4fac4fff616461d9e0b116e98bdd5e8 [id=b82dad04c5514172867f93c52cb39722]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.029536 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527375706fef48fda1adeeddf2434cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.030129 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c3d9f992800e4a41a7af7e2751214eaa (DEFAULT_TABLE table=default.Table_5cf48a44b13a4059b1e8974875e736e2 [id=7080faa7b3844702859c8a48be032b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.030426 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d9f992800e4a41a7af7e2751214eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.033432 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36e80f0671f404bbd0e2dded93716a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.034332 753 tablet_service.cc:1467] Processing CreateTablet for tablet d36e80f0671f404bbd0e2dded93716a3 (DEFAULT_TABLE table=default.Table_486f2f72d50e49cdb7e8a1c022cb439d [id=475d752e44c14cf7bdf3d447356f0c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.035033 1160 ts_tablet_manager.cc:1918] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.035207 1160 log.cc:1198] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ea5d462230734767bb010f520c86c5ab
I20250109 20:52:58.033926 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 527375706fef48fda1adeeddf2434cb5 (DEFAULT_TABLE table=default.Table_a4fac4fff616461d9e0b116e98bdd5e8 [id=b82dad04c5514172867f93c52cb39722]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.036376 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527375706fef48fda1adeeddf2434cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.037181 896 tablet_service.cc:1467] Processing CreateTablet for tablet c3d9f992800e4a41a7af7e2751214eaa (DEFAULT_TABLE table=default.Table_5cf48a44b13a4059b1e8974875e736e2 [id=7080faa7b3844702859c8a48be032b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.037496 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d9f992800e4a41a7af7e2751214eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.037513 740 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3a253685e34e80a26f338be66d31bb (DEFAULT_TABLE table=default.Table_17e950d4133f4d2a8eeab7f7af0b10c2 [id=e954185bc21e421f824d516f2cf4285b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.037788 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3a253685e34e80a26f338be66d31bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.038941 1160 ts_tablet_manager.cc:1939] T ea5d462230734767bb010f520c86c5ab P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:58.039604 1160 tablet_replica.cc:331] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:58.039793 1160 raft_consensus.cc:2238] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.039958 1160 raft_consensus.cc:2267] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.040616 1160 ts_tablet_manager.cc:1905] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.042016 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ea5d462230734767bb010f520c86c5ab (table default.Table_2cb493640d614e088e638e580754f9d4 [id=d7d515f88f8b4c70873fbf17f29bb622]) successfully deleted
W20250109 20:52:58.042299 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet bcd3219734314ae1ad03968fdf9c9cd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcd3219734314ae1ad03968fdf9c9cd0 already in progress: creating tablet
W20250109 20:52:58.042531 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 76bd758fa2fa4cebbdf7faf3856f1941 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76bd758fa2fa4cebbdf7faf3856f1941 already in progress: creating tablet
I20250109 20:52:58.043826 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3a253685e34e80a26f338be66d31bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.051597 890 tablet_service.cc:1467] Processing CreateTablet for tablet 156fbcdb4c9b4f83a96dad732a9a0964 (DEFAULT_TABLE table=default.Table_27dce06a871149eb94e251f6e60e048d [id=48b6cb16548444bab4eb69d94cc3da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.052492 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156fbcdb4c9b4f83a96dad732a9a0964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.054327 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36e80f0671f404bbd0e2dded93716a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.058099 1176 tablet_bootstrap.cc:654] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.059026 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5b107b4c26a94898b744654195bfe567
I20250109 20:52:58.059518 447 catalog_manager.cc:6596] ReplaceTablet: tablet 5b107b4c26a94898b744654195bfe567 deleted and replaced by tablet 670375bfd5114040aed6be25d1374d42
I20250109 20:52:58.060218 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b107b4c26a94898b744654195bfe567 with delete_type TABLET_DATA_DELETED (replaced by tablet 670375bfd5114040aed6be25d1374d42) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.062029 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b107b4c26a94898b744654195bfe567 with delete_type TABLET_DATA_DELETED (replaced by tablet 670375bfd5114040aed6be25d1374d42) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.063338 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f8eba3dcab764a98857604b63bf9dce1
I20250109 20:52:58.063882 880 tablet_service.cc:1514] Processing DeleteTablet for tablet f8eba3dcab764a98857604b63bf9dce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.064357 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f8eba3dcab764a98857604b63bf9dce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.064404 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f8eba3dcab764a98857604b63bf9dce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:58.065205 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e233caae7679488d9b61f34e5f57b4d3 has no consensus state
I20250109 20:52:58.067301 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_beeef07faf3a416aa68ffed7aab3a7b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.067883 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_beeef07faf3a416aa68ffed7aab3a7b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.069082 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1a922ef3992c4ec088cfdf98b527dcf2" } new_table_name: "default.Table_628c3b9194114d9d920428c59ced645e" modify_external_catalogs: true
I20250109 20:52:58.071130 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 9558dfe563d94482bf496e19b8a80edc (table default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6]). Expected version 2 got 0
I20250109 20:52:58.071615 1975 consensus_queue.cc:1035] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:58.072890 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fe13baff8c5840dea64e9e438c5e2e49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.073326 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe13baff8c5840dea64e9e438c5e2e49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.074733 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4b5c0e0e5bfa4599ab2035fa66341d45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:58.074977 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b107b4c26a94898b744654195bfe567 with delete_type TABLET_DATA_DELETED (replaced by tablet 670375bfd5114040aed6be25d1374d42) from {username='slave'} at 127.0.0.1:52040
W20250109 20:52:58.075373 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b5c0e0e5bfa4599ab2035fa66341d45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.076921 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_676e3bec2c46486b82b8e636ba06a4d3" } modify_external_catalogs: true
I20250109 20:52:58.077142 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_676e3bec2c46486b82b8e636ba06a4d3" } modify_external_catalogs: true
I20250109 20:52:58.080456 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 9558dfe563d94482bf496e19b8a80edc (table default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6]). Expected version 2 got 0
I20250109 20:52:58.081616 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9bebcb16a96c49088a5ac609ee22c5fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.082057 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bebcb16a96c49088a5ac609ee22c5fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.088797 1975 raft_consensus.cc:491] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.089000 1975 raft_consensus.cc:513] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.089766 1975 leader_election.cc:290] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.090260 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b083e6be42f41ecbf23ee57ec17b0a5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:58.090502 1051 raft_consensus.cc:2463] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:58.090884 713 leader_election.cc:304] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:58.091159 1975 raft_consensus.cc:2798] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:58.091244 1975 raft_consensus.cc:491] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:58.091320 1975 raft_consensus.cc:3054] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:58.092993 1678 raft_consensus.cc:491] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.093101 1678 raft_consensus.cc:513] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.093276 753 tablet_service.cc:1467] Processing CreateTablet for tablet 156fbcdb4c9b4f83a96dad732a9a0964 (DEFAULT_TABLE table=default.Table_27dce06a871149eb94e251f6e60e048d [id=48b6cb16548444bab4eb69d94cc3da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.093679 1678 leader_election.cc:290] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.093715 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156fbcdb4c9b4f83a96dad732a9a0964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.093611 1975 raft_consensus.cc:513] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:58.095090 713 leader_election.cc:343] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 34f215eee55d4e6a97a384df6ff9d596
I20250109 20:52:58.095094 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 156fbcdb4c9b4f83a96dad732a9a0964 (DEFAULT_TABLE table=default.Table_27dce06a871149eb94e251f6e60e048d [id=48b6cb16548444bab4eb69d94cc3da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.095678 1975 leader_election.cc:290] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.095736 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b083e6be42f41ecbf23ee57ec17b0a5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:58.095922 1051 raft_consensus.cc:3054] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:58.096673 887 tablet_service.cc:1467] Processing CreateTablet for tablet 1343e8e822684fea8a13d8df9ec33779 (DEFAULT_TABLE table=default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.097054 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1343e8e822684fea8a13d8df9ec33779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.097684 880 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1499e571354faf9952af5288b5c3fa (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.098325 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1499e571354faf9952af5288b5c3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.098824 750 tablet_service.cc:1467] Processing CreateTablet for tablet f8232ec65c5f499896e68d118b3c4d79 (DEFAULT_TABLE table=default.Table_d9ef5e30a4624d08b31b37a6c230d437 [id=b98e5248d11b4a878d730d1e1ad5b7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.099066 1051 raft_consensus.cc:2463] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:58.099134 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8232ec65c5f499896e68d118b3c4d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.099143 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f8232ec65c5f499896e68d118b3c4d79 (DEFAULT_TABLE table=default.Table_d9ef5e30a4624d08b31b37a6c230d437 [id=b98e5248d11b4a878d730d1e1ad5b7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.095908 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 8971ca3e32dd479e928709e3a6f6b2f1 (DEFAULT_TABLE table=default.Table_ed7be8358b1c4e17acb5190ca02be11b [id=8ff9fe7726814729a1764caf1d1587fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.099390 713 leader_election.cc:304] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:58.099413 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8232ec65c5f499896e68d118b3c4d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.103612 756 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4d5613811444c38ea786a6ad43f9c1 (DEFAULT_TABLE table=default.Table_2d8d4a28c8d342e7be15b81a4a9785ae [id=c735b8dfead74ac69203599bf319f6b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.103873 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4d5613811444c38ea786a6ad43f9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.105168 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8971ca3e32dd479e928709e3a6f6b2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.108032 1975 raft_consensus.cc:2798] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:58.108955 1975 raft_consensus.cc:695] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:58.109292 1975 consensus_queue.cc:237] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.112377 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 1343e8e822684fea8a13d8df9ec33779 (DEFAULT_TABLE table=default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.113029 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1499e571354faf9952af5288b5c3fa (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.113463 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1343e8e822684fea8a13d8df9ec33779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.117000 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1499e571354faf9952af5288b5c3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.118690 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1499e571354faf9952af5288b5c3fa (DEFAULT_TABLE table=default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.119021 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1499e571354faf9952af5288b5c3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.120972 739 tablet_service.cc:1467] Processing CreateTablet for tablet 1343e8e822684fea8a13d8df9ec33779 (DEFAULT_TABLE table=default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.121258 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1343e8e822684fea8a13d8df9ec33779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.123836 1678 raft_consensus.cc:491] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.123943 1678 raft_consensus.cc:513] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.124495 1678 leader_election.cc:290] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.124604 1678 consensus_queue.cc:1035] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:58.123481 886 tablet_service.cc:1467] Processing CreateTablet for tablet 8971ca3e32dd479e928709e3a6f6b2f1 (DEFAULT_TABLE table=default.Table_ed7be8358b1c4e17acb5190ca02be11b [id=8ff9fe7726814729a1764caf1d1587fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:58.127344 713 leader_election.cc:343] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: fbe8a9e5b5fe406b988126cd1859eabc
I20250109 20:52:58.134989 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8971ca3e32dd479e928709e3a6f6b2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.136904 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156fbcdb4c9b4f83a96dad732a9a0964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.137253 896 tablet_service.cc:1467] Processing CreateTablet for tablet f8232ec65c5f499896e68d118b3c4d79 (DEFAULT_TABLE table=default.Table_d9ef5e30a4624d08b31b37a6c230d437 [id=b98e5248d11b4a878d730d1e1ad5b7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.140867 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4f20d7d11abf4421aa8bc8467e69cc37" } new_table_name: "default.Table_c2ad7ac382e141cdbf0e04072bc26ffe" modify_external_catalogs: true
I20250109 20:52:58.145133 747 tablet_service.cc:1467] Processing CreateTablet for tablet 8971ca3e32dd479e928709e3a6f6b2f1 (DEFAULT_TABLE table=default.Table_ed7be8358b1c4e17acb5190ca02be11b [id=8ff9fe7726814729a1764caf1d1587fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:58.145525 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 420f755e81eb4ddd8b872bb7dcea4d13 has no consensus state
I20250109 20:52:58.145634 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8971ca3e32dd479e928709e3a6f6b2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.146992 448 catalog_manager.cc:5526] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
W20250109 20:52:58.148520 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 643e337667d34534b88715704ab88e82 consensus state has no leader
I20250109 20:52:58.149740 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4560626d5c8547678952bb1e7da86ad4" } modify_external_catalogs: true
I20250109 20:52:58.149943 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4560626d5c8547678952bb1e7da86ad4" } modify_external_catalogs: true
I20250109 20:52:58.150511 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4d5613811444c38ea786a6ad43f9c1 (DEFAULT_TABLE table=default.Table_2d8d4a28c8d342e7be15b81a4a9785ae [id=c735b8dfead74ac69203599bf319f6b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.150925 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4d5613811444c38ea786a6ad43f9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.155880 740 tablet_service.cc:1467] Processing CreateTablet for tablet 7204d58f2cf640059ad827e11f62912a (DEFAULT_TABLE table=default.Table_6dde16e3825445a38efe43db92927674 [id=4ad71ffbd3b5478087f0cd6b3a3f7bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.156035 1176 tablet_bootstrap.cc:492] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.156188 1176 ts_tablet_manager.cc:1397] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.003s
I20250109 20:52:58.157335 1176 raft_consensus.cc:357] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.159399 1176 raft_consensus.cc:383] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.159479 1176 raft_consensus.cc:738] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.159731 1176 consensus_queue.cc:260] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.160410 1176 ts_tablet_manager.cc:1428] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:58.160745 1176 tablet_bootstrap.cc:492] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.163551 1160 ts_tablet_manager.cc:1918] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.156370 1975 raft_consensus.cc:491] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.166287 1975 raft_consensus.cc:513] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.166824 1975 leader_election.cc:290] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:58.167586 713 leader_election.cc:343] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: b405d16cc62444f2ad94fb69bb9fb6ee
I20250109 20:52:58.167986 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7204d58f2cf640059ad827e11f62912a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.167951 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 7204d58f2cf640059ad827e11f62912a (DEFAULT_TABLE table=default.Table_6dde16e3825445a38efe43db92927674 [id=4ad71ffbd3b5478087f0cd6b3a3f7bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.168344 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7204d58f2cf640059ad827e11f62912a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.173981 1960 raft_consensus.cc:491] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.174129 1960 raft_consensus.cc:513] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.174649 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2ddb127ac2ad49b9820a91a0f6b010a8
I20250109 20:52:58.174877 1960 leader_election.cc:290] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.175536 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728e9b6101e3447fb3c24b3917aa4872" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:58.176188 990 leader_election.cc:343] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.176632 1176 tablet_bootstrap.cc:654] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.177491 1960 raft_consensus.cc:491] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.177621 1960 raft_consensus.cc:513] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.177825 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.178253 1960 leader_election.cc:290] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.178766 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4f047e8fa34ca79e78915c61a8ba12" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:58.179294 990 leader_election.cc:343] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.176851 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.181450 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2f6147aaca5c4410a3e4aedb6a916667" } modify_external_catalogs: true
I20250109 20:52:58.181627 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2f6147aaca5c4410a3e4aedb6a916667" } modify_external_catalogs: true
I20250109 20:52:58.183109 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fe23a0c1fdf142f5a7297a0f35955a50" } new_table_name: "default.Table_d8cd5c02957d4036b34c6891b99feee8" modify_external_catalogs: true
I20250109 20:52:58.203248 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_69d6ced747db4140aee267f44cad8d13" } modify_external_catalogs: true
W20250109 20:52:58.209640 712 consensus_peers.cc:487] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:52:58.213951 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ffd3a1b1d3e44feda6dbaa8556e377fc (table default.Table_b856f34058994714afb4ce41d44f5384 [id=7adda20a8381463a9f6decd4c7cd045d]) successfully deleted
I20250109 20:52:58.214250 1166 tablet_replica.cc:331] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.214483 1166 raft_consensus.cc:2238] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.214718 1166 raft_consensus.cc:2267] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.215437 1166 ts_tablet_manager.cc:1905] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.219062 1922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250109 20:52:58.220032 1922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250109 20:52:58.220309 1922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250109 20:52:58.221566 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_69d6ced747db4140aee267f44cad8d13" } modify_external_catalogs: true
I20250109 20:52:58.223104 1166 ts_tablet_manager.cc:1918] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.223213 1166 log.cc:1198] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2a4f047e8fa34ca79e78915c61a8ba12
I20250109 20:52:58.223498 1166 ts_tablet_manager.cc:1939] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.224079 1176 tablet_bootstrap.cc:492] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.224210 1176 ts_tablet_manager.cc:1397] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.000s
I20250109 20:52:58.224136 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2a4f047e8fa34ca79e78915c61a8ba12 (table default.Table_70499a7567fa4d6498705309cde3b257 [id=3ae741f5fe4343188b6f0d10b2a09b69]) successfully deleted
I20250109 20:52:58.224449 1166 tablet_replica.cc:331] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.224498 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3]). Expected version 2 got 0
I20250109 20:52:58.224601 1166 raft_consensus.cc:2238] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.224759 1166 raft_consensus.cc:2267] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.224865 1176 raft_consensus.cc:357] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.225068 1176 raft_consensus.cc:383] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.225134 1176 raft_consensus.cc:738] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.225322 1176 consensus_queue.cc:260] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.225421 1166 ts_tablet_manager.cc:1905] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.226011 1176 ts_tablet_manager.cc:1428] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:58.226545 1176 tablet_bootstrap.cc:492] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.227998 1176 tablet_bootstrap.cc:654] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:58.235921 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c8d8d771b56407d94fcb23d76796039 consensus state has no leader
I20250109 20:52:58.239426 1166 ts_tablet_manager.cc:1918] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.239604 1166 log.cc:1198] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/728e9b6101e3447fb3c24b3917aa4872
I20250109 20:52:58.239984 1166 ts_tablet_manager.cc:1939] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.240620 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 728e9b6101e3447fb3c24b3917aa4872 (table default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]) successfully deleted
I20250109 20:52:58.240969 1166 tablet_replica.cc:331] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.241197 1166 raft_consensus.cc:2238] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.241428 1166 raft_consensus.cc:2267] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.242056 1166 ts_tablet_manager.cc:1905] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.246659 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 825220629faa4d2481e0db9fa226c306
I20250109 20:52:58.247666 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 825220629faa4d2481e0db9fa226c306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.247999 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 825220629faa4d2481e0db9fa226c306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.248912 1166 ts_tablet_manager.cc:1918] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:58.249012 449 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 8ed134863bf642e79c26e5bf75b248b0 (table default.Table_6640cef9559a4ba18d06c0aed445852f [id=46c011650d964e26b40be8a854e1f8c0]). Expected version 2 got 0
I20250109 20:52:58.249086 1166 log.cc:1198] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c00aaf81b34445abb7502700f01d4f3f
I20250109 20:52:58.249212 449 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3]). Expected version 2 got 0
I20250109 20:52:58.249461 1166 ts_tablet_manager.cc:1939] T c00aaf81b34445abb7502700f01d4f3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.250635 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d409409813420689134773c627833b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.251106 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_385822fe7219471bacdd1ad2dbe7d891 [id=0430588a51ed49ae86290eaab54f79dc]) successfully deleted
W20250109 20:52:58.251818 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet bcd3219734314ae1ad03968fdf9c9cd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcd3219734314ae1ad03968fdf9c9cd0 already in progress: creating tablet
W20250109 20:52:58.252457 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 76bd758fa2fa4cebbdf7faf3856f1941 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76bd758fa2fa4cebbdf7faf3856f1941 already in progress: creating tablet
W20250109 20:52:58.253144 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5b107b4c26a94898b744654195bfe567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b107b4c26a94898b744654195bfe567 already in progress: creating tablet
W20250109 20:52:58.253818 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f8eba3dcab764a98857604b63bf9dce1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8eba3dcab764a98857604b63bf9dce1 already in progress: creating tablet
W20250109 20:52:58.255242 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ddb127ac2ad49b9820a91a0f6b010a8 already in progress: creating tablet
W20250109 20:52:58.255879 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 825220629faa4d2481e0db9fa226c306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825220629faa4d2481e0db9fa226c306 already in progress: creating tablet
I20250109 20:52:58.256203 1166 tablet_replica.cc:331] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.256454 1166 raft_consensus.cc:2238] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.256646 1166 raft_consensus.cc:2267] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.257491 1166 ts_tablet_manager.cc:1905] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.259516 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3]). Expected version 2 got 0
I20250109 20:52:58.261518 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 274a58191d00475e8da9ff72dd33524f
I20250109 20:52:58.262530 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 274a58191d00475e8da9ff72dd33524f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.263110 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 274a58191d00475e8da9ff72dd33524f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.263813 1166 ts_tablet_manager.cc:1918] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.263973 1166 log.cc:1198] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/93d409409813420689134773c627833b
I20250109 20:52:58.264310 1166 ts_tablet_manager.cc:1939] T 93d409409813420689134773c627833b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.265368 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 93d409409813420689134773c627833b (table default.Table_1ee95fba580446b8926cbc779fca720f [id=a4a7c86d0df2412a83f12157c967aae4]) successfully deleted
W20250109 20:52:58.266001 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 274a58191d00475e8da9ff72dd33524f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 274a58191d00475e8da9ff72dd33524f already in progress: creating tablet
I20250109 20:52:58.266039 1157 ts_tablet_manager.cc:1918] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.266357 1157 log.cc:1198] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/8d1630e4438e4337ba026b41f630a480
I20250109 20:52:58.266757 1157 ts_tablet_manager.cc:1939] T 8d1630e4438e4337ba026b41f630a480 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.268014 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 0d81c1734d2440ce9957880c897a4cc1
W20250109 20:52:58.268258 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7075e2a8cf0440b788135f15a182337a consensus state has no leader
I20250109 20:52:58.269970 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 8d1630e4438e4337ba026b41f630a480 (table default.Table_d7539ac3866842c996d4ced88e764877 [id=b78805ca41e647e1927d2564fb00b21f]) successfully deleted
I20250109 20:52:58.270363 1157 tablet_replica.cc:331] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:58.270593 1157 raft_consensus.cc:2238] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.270745 1157 raft_consensus.cc:2267] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.271312 1960 raft_consensus.cc:491] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.271427 1960 raft_consensus.cc:513] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.271600 1157 ts_tablet_manager.cc:1905] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.272006 1960 leader_election.cc:290] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.168550 899 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4d5613811444c38ea786a6ad43f9c1 (DEFAULT_TABLE table=default.Table_2d8d4a28c8d342e7be15b81a4a9785ae [id=c735b8dfead74ac69203599bf319f6b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.272867 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed134863bf642e79c26e5bf75b248b0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:58.168901 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7204d58f2cf640059ad827e11f62912a (DEFAULT_TABLE table=default.Table_6dde16e3825445a38efe43db92927674 [id=4ad71ffbd3b5478087f0cd6b3a3f7bbf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:58.273283 990 leader_election.cc:343] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.169258 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8232ec65c5f499896e68d118b3c4d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.192067 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.276918 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed134863bf642e79c26e5bf75b248b0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.277431 989 leader_election.cc:343] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.277537 989 leader_election.cc:304] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.277671 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7204d58f2cf640059ad827e11f62912a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.277967 1960 raft_consensus.cc:2743] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.279821 1160 log.cc:1198] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2d96a569e6be4e01a26918d115cfbcf9
I20250109 20:52:58.280099 1160 ts_tablet_manager.cc:1939] T 2d96a569e6be4e01a26918d115cfbcf9 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:58.280558 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2d96a569e6be4e01a26918d115cfbcf9 (table default.Table_ff7c9fe449d540f19d0ce42d44ba0b73 [id=028851fe80724d3cb4402101784bd12a]) successfully deleted
W20250109 20:52:58.281071 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5b107b4c26a94898b744654195bfe567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b107b4c26a94898b744654195bfe567 already in progress: creating tablet
I20250109 20:52:58.281318 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4d5613811444c38ea786a6ad43f9c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:58.281631 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f8eba3dcab764a98857604b63bf9dce1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8eba3dcab764a98857604b63bf9dce1 already in progress: creating tablet
W20250109 20:52:58.282096 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ddb127ac2ad49b9820a91a0f6b010a8 already in progress: creating tablet
I20250109 20:52:58.175315 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728e9b6101e3447fb3c24b3917aa4872" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:58.193285 1165 tablet_bootstrap.cc:492] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:58.285373 1165 ts_tablet_manager.cc:1397] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.372s user 0.003s sys 0.000s
I20250109 20:52:58.286504 1165 raft_consensus.cc:357] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.286650 1165 raft_consensus.cc:383] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.286687 1165 raft_consensus.cc:738] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.286852 1165 consensus_queue.cc:260] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.178632 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4f047e8fa34ca79e78915c61a8ba12" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:58.197911 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b083e6be42f41ecbf23ee57ec17b0a5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:58.207036 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f215eee55d4e6a97a384df6ff9d596" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:58.207244 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe8a9e5b5fe406b988126cd1859eabc" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.287312 989 leader_election.cc:343] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.207635 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b083e6be42f41ecbf23ee57ec17b0a5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:58.287407 989 leader_election.cc:304] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:58.287567 989 raft_consensus.cc:2681] T 728e9b6101e3447fb3c24b3917aa4872 P 5fcc647ed479491b88e5ffacf2927a1a: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250109 20:52:58.287715 712 leader_election.cc:343] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:58.287988 712 leader_election.cc:343] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.288065 712 leader_election.cc:304] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:52:58.288108 989 leader_election.cc:343] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.288185 989 leader_election.cc:304] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:58.288303 989 raft_consensus.cc:2681] T 2a4f047e8fa34ca79e78915c61a8ba12 P 5fcc647ed479491b88e5ffacf2927a1a: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250109 20:52:58.288363 712 leader_election.cc:343] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.288435 712 leader_election.cc:304] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.208364 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b405d16cc62444f2ad94fb69bb9fb6ee" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:58.288941 1984 raft_consensus.cc:2743] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.289101 1984 raft_consensus.cc:2743] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.290680 1165 ts_tablet_manager.cc:1428] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:58.290913 1165 tablet_bootstrap.cc:492] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:52:58.291568 712 leader_election.cc:343] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:58.291841 712 leader_election.cc:343] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.291918 712 leader_election.cc:304] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.292187 1678 raft_consensus.cc:2743] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.262809 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 274a58191d00475e8da9ff72dd33524f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.292436 1165 tablet_bootstrap.cc:654] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.292526 1176 tablet_bootstrap.cc:492] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.292635 1176 ts_tablet_manager.cc:1397] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.066s user 0.017s sys 0.001s
I20250109 20:52:58.293368 1176 raft_consensus.cc:357] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.293541 1176 raft_consensus.cc:383] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.293589 1176 raft_consensus.cc:738] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.293802 1176 consensus_queue.cc:260] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.270933 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 825220629faa4d2481e0db9fa226c306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.297338 1176 ts_tablet_manager.cc:1428] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:58.297406 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
I20250109 20:52:58.297561 1176 tablet_bootstrap.cc:492] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.299396 1176 tablet_bootstrap.cc:654] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.301146 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
W20250109 20:52:58.305925 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6e020eeb03e041ed977779a07de5bb80 (table default.Table_1f4279f07e8b4e0a86928bdb9a67bbf6 [id=8bec14b8b4b74d9c87fe86950f5128e1], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6e020eeb03e041ed977779a07de5bb80 consensus state has no leader
I20250109 20:52:58.307024 1176 tablet_bootstrap.cc:492] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.308929 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e56f30e6f3450a8426cd40db89e009 with delete_type TABLET_DATA_DELETED (replaced by tablet eaea4f239cce41a997496a1c94810ac9) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.309284 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb" } modify_external_catalogs: true
I20250109 20:52:58.309525 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb" } modify_external_catalogs: true
I20250109 20:52:58.309407 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dc20220cc040480385a78533a6fe34e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.310065 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc20220cc040480385a78533a6fe34e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.311167 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_26ccdfe61e2a4a7f9afbb1738ea4d483" } new_table_name: "default.Table_0504dfd389e44bafa7837757c1cec711" modify_external_catalogs: true
I20250109 20:52:58.311614 1166 tablet_replica.cc:331] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.311908 1166 raft_consensus.cc:2238] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.312126 1166 raft_consensus.cc:2267] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.312245 1176 ts_tablet_manager.cc:1397] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20250109 20:52:58.312354 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5b016d79eedf43bbbcdc6d742dde478e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.312819 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b016d79eedf43bbbcdc6d742dde478e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.312901 1166 ts_tablet_manager.cc:1905] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.312949 1176 raft_consensus.cc:357] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.313078 1176 raft_consensus.cc:383] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.313128 1176 raft_consensus.cc:738] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.313302 1176 consensus_queue.cc:260] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.314026 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d7b0c3e0c004488b982f368831954313"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.314399 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7b0c3e0c004488b982f368831954313 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.315783 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f60b81fe14f9450f8d5b0885834f3dd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.316206 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f60b81fe14f9450f8d5b0885834f3dd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.316334 1158 raft_consensus.cc:383] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.316424 1158 raft_consensus.cc:738] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.317523 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_48fcb713e28c46928c9308b50268e442"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.318027 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48fcb713e28c46928c9308b50268e442 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.318135 1176 ts_tablet_manager.cc:1428] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:52:58.320333 1176 tablet_bootstrap.cc:492] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:58.322307 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 274a58191d00475e8da9ff72dd33524f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 274a58191d00475e8da9ff72dd33524f already in progress: creating tablet
W20250109 20:52:58.322721 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 825220629faa4d2481e0db9fa226c306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825220629faa4d2481e0db9fa226c306 already in progress: creating tablet
I20250109 20:52:58.330055 1176 tablet_bootstrap.cc:654] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.332204 1157 ts_tablet_manager.cc:1918] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.336349 1176 tablet_bootstrap.cc:492] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.336493 1176 ts_tablet_manager.cc:1397] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20250109 20:52:58.336517 1157 log.cc:1198] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/72a455033b674e5a94fc37d0275eb313
I20250109 20:52:58.337170 1176 raft_consensus.cc:357] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.337761 1176 raft_consensus.cc:383] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.337837 1176 raft_consensus.cc:738] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.337805 1157 ts_tablet_manager.cc:1939] T 72a455033b674e5a94fc37d0275eb313 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.338027 1176 consensus_queue.cc:260] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.339221 1176 ts_tablet_manager.cc:1428] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:58.339279 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 72a455033b674e5a94fc37d0275eb313 (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]) successfully deleted
I20250109 20:52:58.339601 1158 consensus_queue.cc:260] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.339756 1176 tablet_bootstrap.cc:492] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.342546 1176 tablet_bootstrap.cc:654] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.344403 1166 ts_tablet_manager.cc:1918] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.344483 1166 log.cc:1198] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/57e56f30e6f3450a8426cd40db89e009
I20250109 20:52:58.344805 1166 ts_tablet_manager.cc:1939] T 57e56f30e6f3450a8426cd40db89e009 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.346098 1158 ts_tablet_manager.cc:1428] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.349s user 0.015s sys 0.003s
I20250109 20:52:58.347045 1158 tablet_bootstrap.cc:492] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.348562 1158 tablet_bootstrap.cc:654] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:58.349892 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cca642fb9c3e4a55adfb6ad8386a67cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cca642fb9c3e4a55adfb6ad8386a67cd already in progress: creating tablet
I20250109 20:52:58.350473 1157 tablet_replica.cc:331] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:58.350703 1157 raft_consensus.cc:2238] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:58.351019 1157 pending_rounds.cc:62] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Trying to abort 1 pending ops.
I20250109 20:52:58.351120 1157 pending_rounds.cc:69] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523684630016000 op_type: NO_OP noop_request { }
I20250109 20:52:58.351310 1157 raft_consensus.cc:2883] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:52:58.351440 1157 raft_consensus.cc:2267] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.354112 1176 tablet_bootstrap.cc:492] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.354234 1176 ts_tablet_manager.cc:1397] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20250109 20:52:58.354508 1157 ts_tablet_manager.cc:1905] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.354480 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 57e56f30e6f3450a8426cd40db89e009 (table default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]) successfully deleted
I20250109 20:52:58.354954 1176 raft_consensus.cc:357] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.355141 1176 raft_consensus.cc:383] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.355186 1176 raft_consensus.cc:738] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.356909 1176 consensus_queue.cc:260] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.357715 1960 raft_consensus.cc:491] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.357822 1960 raft_consensus.cc:513] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.358533 1960 leader_election.cc:290] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.359140 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d09daa2eed41c3952fa21aa2cdd09e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:58.359318 1176 ts_tablet_manager.cc:1428] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:58.359539 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d09daa2eed41c3952fa21aa2cdd09e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.360193 990 leader_election.cc:343] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:58.360337 989 leader_election.cc:343] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.360435 989 leader_election.cc:304] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.360878 1960 raft_consensus.cc:2743] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.361191 1176 tablet_bootstrap.cc:492] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.362619 1176 tablet_bootstrap.cc:654] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.368255 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_77c9fc51089f43c0a1c2db8a4d77573f" } new_table_name: "default.Table_77fce37f901d442a85f1892961ef0325" modify_external_catalogs: true
I20250109 20:52:58.368556 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5533e631d0dc4529b038c22187a08afd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:58.369369 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9f92e5ee4b5b4f7fa2ec542cc8ace5f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:58.370527 1176 tablet_bootstrap.cc:492] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.370636 1176 ts_tablet_manager.cc:1397] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20250109 20:52:58.371110 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6f6d795dd7a7434c9f4fb1e068e36849" } new_table_name: "default.Table_b1963e7eee964bdca86d4a27b5b2d063" modify_external_catalogs: true
I20250109 20:52:58.371279 1176 raft_consensus.cc:357] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.371441 1176 raft_consensus.cc:383] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.371490 1176 raft_consensus.cc:738] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.373093 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f3adc0bbe9994d269df1e03dd5541e33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:58.373163 1176 consensus_queue.cc:260] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.374142 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ba4ca37f1429486eb5410290edb271e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:58.374812 1176 ts_tablet_manager.cc:1428] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:58.376065 1176 tablet_bootstrap.cc:492] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.376236 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ca2b022ad9274c0e9b67560389d8dbed
I20250109 20:52:58.378017 1176 tablet_bootstrap.cc:654] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.382227 1158 tablet_bootstrap.cc:492] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.382352 1158 ts_tablet_manager.cc:1397] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250109 20:52:58.382967 1158 raft_consensus.cc:357] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.383154 1158 raft_consensus.cc:383] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.383198 1158 raft_consensus.cc:738] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.398069 1157 ts_tablet_manager.cc:1918] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:58.398960 1157 log.cc:1198] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/af8c0bf27e994e6aa4cbedca16febe12
I20250109 20:52:58.402073 1157 ts_tablet_manager.cc:1939] T af8c0bf27e994e6aa4cbedca16febe12 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.405339 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_de7dc1f873f44dcb9e4eb13f73a25657" } modify_external_catalogs: true
I20250109 20:52:58.405556 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_de7dc1f873f44dcb9e4eb13f73a25657" } modify_external_catalogs: true
I20250109 20:52:58.406589 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet af8c0bf27e994e6aa4cbedca16febe12 (table default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]) successfully deleted
I20250109 20:52:58.407403 1157 tablet_replica.cc:331] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:58.408317 1157 raft_consensus.cc:2238] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.410925 1157 raft_consensus.cc:2267] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.414882 1157 ts_tablet_manager.cc:1905] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:58.416814 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3adc0bbe9994d269df1e03dd5541e33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:58.417007 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5533e631d0dc4529b038c22187a08afd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:58.418592 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba4ca37f1429486eb5410290edb271e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.419988 740 tablet_service.cc:1514] Processing DeleteTablet for tablet ca2b022ad9274c0e9b67560389d8dbed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.420900 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ca2b022ad9274c0e9b67560389d8dbed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.421160 1166 tablet_replica.cc:331] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:52:58.421298 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f92e5ee4b5b4f7fa2ec542cc8ace5f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.421401 1166 raft_consensus.cc:2238] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.421576 1166 raft_consensus.cc:2267] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.422356 1166 ts_tablet_manager.cc:1905] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.423511 886 tablet_service.cc:1514] Processing DeleteTablet for tablet ca2b022ad9274c0e9b67560389d8dbed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.423909 1158 consensus_queue.cc:260] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:58.424149 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ca2b022ad9274c0e9b67560389d8dbed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca2b022ad9274c0e9b67560389d8dbed already in progress: creating tablet
I20250109 20:52:58.426172 1158 ts_tablet_manager.cc:1428] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20250109 20:52:58.426445 1158 tablet_bootstrap.cc:492] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.428369 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_019702682d2a460fab654ee8e4fc3463"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.428958 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_019702682d2a460fab654ee8e4fc3463 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.429243 1166 ts_tablet_manager.cc:1918] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:58.429394 1166 log.cc:1198] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ca2b022ad9274c0e9b67560389d8dbed
I20250109 20:52:58.429653 1158 tablet_bootstrap.cc:654] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.430038 1166 ts_tablet_manager.cc:1939] T ca2b022ad9274c0e9b67560389d8dbed P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.431166 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ca2b022ad9274c0e9b67560389d8dbed (table default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb [id=5d0cc83d6e3e4da4845ec153a35cd890]) successfully deleted
I20250109 20:52:58.439291 2125 tablet.cc:1721] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250109 20:52:58.441095 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4f6196d820f4aecaff2701aaf56faf5 consensus state has no leader
W20250109 20:52:58.441593 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd92f4c7abea454c8d68c1f5023d5727 consensus state has no leader
I20250109 20:52:58.441809 1157 ts_tablet_manager.cc:1918] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.442034 1157 log.cc:1198] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/fbe8a9e5b5fe406b988126cd1859eabc
I20250109 20:52:58.442433 1157 ts_tablet_manager.cc:1939] T fbe8a9e5b5fe406b988126cd1859eabc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.443715 1157 tablet_replica.cc:331] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:58.443903 2126 tablet.cc:1721] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:58.443986 1157 raft_consensus.cc:2238] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.444247 1157 raft_consensus.cc:2267] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.444846 1960 raft_consensus.cc:491] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.444968 1960 raft_consensus.cc:513] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.445534 1960 leader_election.cc:290] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.446180 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e9b31b27949c2948cb64b6124ee72" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:58.446615 990 leader_election.cc:343] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.447072 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e9b31b27949c2948cb64b6124ee72" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.447935 989 leader_election.cc:343] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.448027 989 leader_election.cc:304] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.448774 1960 raft_consensus.cc:2743] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.450044 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet fbe8a9e5b5fe406b988126cd1859eabc (table default.Table_d4b677aa33534a4d8261f52878181252 [id=2f89d33acf544985bafb538d400c0d11]) successfully deleted
I20250109 20:52:58.451614 1157 ts_tablet_manager.cc:1905] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.458067 1157 ts_tablet_manager.cc:1918] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.458233 1157 log.cc:1198] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/34f215eee55d4e6a97a384df6ff9d596
I20250109 20:52:58.458547 1157 ts_tablet_manager.cc:1939] T 34f215eee55d4e6a97a384df6ff9d596 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.459440 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 34f215eee55d4e6a97a384df6ff9d596 (table default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]) successfully deleted
I20250109 20:52:58.459870 1157 tablet_replica.cc:331] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:58.460062 1157 raft_consensus.cc:2238] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.460238 1157 raft_consensus.cc:2267] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.460884 1157 ts_tablet_manager.cc:1905] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.461311 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f20230357dc241449b6b6dc5f5479105" } new_table_name: "default.Table_ca51464e265c4e28b9e54fe7be0587fc" modify_external_catalogs: true
I20250109 20:52:58.462443 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4f4640c9222649fc957636279628a391" } modify_external_catalogs: true
I20250109 20:52:58.462610 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4f4640c9222649fc957636279628a391" } modify_external_catalogs: true
I20250109 20:52:58.464334 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ba4ca37f1429486eb5410290edb271e2" } modify_external_catalogs: true
I20250109 20:52:58.464646 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ba4ca37f1429486eb5410290edb271e2" } modify_external_catalogs: true
I20250109 20:52:58.466737 1165 tablet_bootstrap.cc:492] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:58.466912 1165 ts_tablet_manager.cc:1397] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.176s user 0.011s sys 0.003s
I20250109 20:52:58.466976 2130 tablet.cc:1721] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:58.470495 1157 ts_tablet_manager.cc:1918] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.470661 1157 log.cc:1198] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/b405d16cc62444f2ad94fb69bb9fb6ee
I20250109 20:52:58.471078 1157 ts_tablet_manager.cc:1939] T b405d16cc62444f2ad94fb69bb9fb6ee P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.472915 1165 raft_consensus.cc:357] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.473670 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9db2524d4be648e4828df22a952c1cb4" } modify_external_catalogs: true
I20250109 20:52:58.473785 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet b405d16cc62444f2ad94fb69bb9fb6ee (table default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]) successfully deleted
I20250109 20:52:58.474339 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9db2524d4be648e4828df22a952c1cb4" } modify_external_catalogs: true
I20250109 20:52:58.475293 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5a6fe1039e374d2bb76ad9117530c055" } new_table_name: "default.Table_6c787d10220c41dca63172c9b3bf70ae" modify_external_catalogs: true
I20250109 20:52:58.477669 1157 tablet_replica.cc:331] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:58.478713 1157 raft_consensus.cc:2238] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.478986 1157 raft_consensus.cc:2267] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:58.479591 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b1891cbef8f4b0cae0c15d478db4860 consensus state has no leader
I20250109 20:52:58.479591 1157 ts_tablet_manager.cc:1905] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.479642 1158 tablet_bootstrap.cc:492] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.479825 1158 ts_tablet_manager.cc:1397] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.001s
W20250109 20:52:58.485092 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14fb1017fe6c4f5abd640ad1f6214232 consensus state has no leader
I20250109 20:52:58.485633 1157 ts_tablet_manager.cc:1918] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:58.485781 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 35cfb602ad38408e8492b88eb78a438c
I20250109 20:52:58.491585 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_0504dfd389e44bafa7837757c1cec711 [id=3e073742d9cf476387b0a0949059d933] alter complete (version 2)
I20250109 20:52:58.492503 1157 log.cc:1198] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/c00aaf81b34445abb7502700f01d4f3f
I20250109 20:52:58.493052 1157 ts_tablet_manager.cc:1939] T c00aaf81b34445abb7502700f01d4f3f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.493994 1165 raft_consensus.cc:383] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.494130 1165 raft_consensus.cc:738] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.494396 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_385822fe7219471bacdd1ad2dbe7d891 [id=0430588a51ed49ae86290eaab54f79dc]) successfully deleted
W20250109 20:52:58.495102 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3008c672bc544000b24b75ae366d9fb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3008c672bc544000b24b75ae366d9fb9 already in progress: creating tablet
I20250109 20:52:58.487835 1158 raft_consensus.cc:357] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.495241 1158 raft_consensus.cc:383] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.495194 1165 consensus_queue.cc:260] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.495362 1158 raft_consensus.cc:738] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:58.495792 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet bcd3219734314ae1ad03968fdf9c9cd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcd3219734314ae1ad03968fdf9c9cd0 already in progress: creating tablet
I20250109 20:52:58.495895 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 9558dfe563d94482bf496e19b8a80edc from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:58.495752 1158 consensus_queue.cc:260] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.497234 1158 ts_tablet_manager.cc:1428] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250109 20:52:58.498042 1165 ts_tablet_manager.cc:1428] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250109 20:52:58.498772 1165 tablet_bootstrap.cc:492] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:58.499537 1158 tablet_bootstrap.cc:492] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.499516 1176 tablet_bootstrap.cc:492] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.499698 1176 ts_tablet_manager.cc:1397] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.124s user 0.019s sys 0.019s
W20250109 20:52:58.500222 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 76bd758fa2fa4cebbdf7faf3856f1941 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76bd758fa2fa4cebbdf7faf3856f1941 already in progress: creating tablet
I20250109 20:52:58.500487 1176 raft_consensus.cc:357] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.501816 1165 tablet_bootstrap.cc:654] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.501886 1158 tablet_bootstrap.cc:654] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.503577 1176 raft_consensus.cc:383] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.503670 1176 raft_consensus.cc:738] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.503892 1176 consensus_queue.cc:260] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.505214 1176 ts_tablet_manager.cc:1428] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:58.505492 1176 tablet_bootstrap.cc:492] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.507011 1176 tablet_bootstrap.cc:654] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.512449 2032 raft_consensus.cc:491] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.512569 2032 raft_consensus.cc:513] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.513422 2032 leader_election.cc:290] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.513938 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7dc6809d6547f6b0354b64d7149bf4" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:58.514464 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7dc6809d6547f6b0354b64d7149bf4" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:58.515352 854 leader_election.cc:304] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.515745 2032 raft_consensus.cc:2743] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.517426 1176 tablet_bootstrap.cc:492] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.517546 1176 ts_tablet_manager.cc:1397] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:52:58.518198 1176 raft_consensus.cc:357] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.518343 1176 raft_consensus.cc:383] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.518379 1176 raft_consensus.cc:738] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.519979 1176 consensus_queue.cc:260] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.521292 1176 ts_tablet_manager.cc:1428] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:58.521806 1176 tablet_bootstrap.cc:492] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.523013 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 7307ebdb0c794451a6d91d23d69fddc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.523237 1160 tablet_replica.cc:331] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:58.523388 1160 raft_consensus.cc:2238] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.523507 1160 raft_consensus.cc:2267] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.524428 1160 ts_tablet_manager.cc:1905] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.526017 1176 tablet_bootstrap.cc:654] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.529131 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c422468931d4ec18c7d8cf9c392d3a6 from {username='slave'} at 127.0.0.1:33768
W20250109 20:52:58.529682 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f8eba3dcab764a98857604b63bf9dce1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8eba3dcab764a98857604b63bf9dce1 already in progress: creating tablet
W20250109 20:52:58.530016 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5b107b4c26a94898b744654195bfe567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b107b4c26a94898b744654195bfe567 already in progress: creating tablet
W20250109 20:52:58.530247 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2ddb127ac2ad49b9820a91a0f6b010a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ddb127ac2ad49b9820a91a0f6b010a8 already in progress: creating tablet
I20250109 20:52:58.530406 1157 tablet_replica.cc:331] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:52:58.530448 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 825220629faa4d2481e0db9fa226c306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825220629faa4d2481e0db9fa226c306 already in progress: creating tablet
I20250109 20:52:58.530570 1157 raft_consensus.cc:2238] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
W20250109 20:52:58.530663 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 274a58191d00475e8da9ff72dd33524f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 274a58191d00475e8da9ff72dd33524f already in progress: creating tablet
I20250109 20:52:58.530834 1157 raft_consensus.cc:2267] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.531701 1157 ts_tablet_manager.cc:1905] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.533313 1160 ts_tablet_manager.cc:1918] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.533496 1160 log.cc:1198] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/7307ebdb0c794451a6d91d23d69fddc6
I20250109 20:52:58.534327 1160 ts_tablet_manager.cc:1939] T 7307ebdb0c794451a6d91d23d69fddc6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:58.535583 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 7307ebdb0c794451a6d91d23d69fddc6 (table default.Table_2e81ac284d7740eb9e743bb21d11a625 [id=6f85f27d20024666b822558de484bc6e]) successfully deleted
I20250109 20:52:58.537539 1176 tablet_bootstrap.cc:492] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.537657 1176 ts_tablet_manager.cc:1397] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250109 20:52:58.538450 1176 raft_consensus.cc:357] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.538643 1176 raft_consensus.cc:383] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.538705 1176 raft_consensus.cc:738] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.541381 1157 ts_tablet_manager.cc:1918] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:52:58.541553 1157 log.cc:1198] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ca2b022ad9274c0e9b67560389d8dbed
I20250109 20:52:58.541977 1157 ts_tablet_manager.cc:1939] T ca2b022ad9274c0e9b67560389d8dbed P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:58.542793 1176 consensus_queue.cc:260] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.543079 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ca2b022ad9274c0e9b67560389d8dbed (table default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb [id=5d0cc83d6e3e4da4845ec153a35cd890]) successfully deleted
I20250109 20:52:58.544234 1176 ts_tablet_manager.cc:1428] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250109 20:52:58.545202 1176 tablet_bootstrap.cc:492] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.546624 2032 raft_consensus.cc:491] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.546736 2032 raft_consensus.cc:513] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.547379 2032 leader_election.cc:290] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.550787 1176 tablet_bootstrap.cc:654] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.552139 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62caa775ef1241f68c0323a2d228c8cd" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:58.552762 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62caa775ef1241f68c0323a2d228c8cd" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:58.553619 1158 tablet_bootstrap.cc:492] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.553755 1158 ts_tablet_manager.cc:1397] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.001s
I20250109 20:52:58.554699 1158 raft_consensus.cc:357] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.554806 854 leader_election.cc:304] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.554922 1158 raft_consensus.cc:383] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.554983 1158 raft_consensus.cc:738] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.555258 2032 raft_consensus.cc:2743] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.555262 1158 consensus_queue.cc:260] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.556592 1158 ts_tablet_manager.cc:1428] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:58.557255 1158 tablet_bootstrap.cc:492] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.558926 1158 tablet_bootstrap.cc:654] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.560377 1165 tablet_bootstrap.cc:492] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:58.560505 1165 ts_tablet_manager.cc:1397] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.062s user 0.015s sys 0.023s
I20250109 20:52:58.560633 1176 tablet_bootstrap.cc:492] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.560762 1176 ts_tablet_manager.cc:1397] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20250109 20:52:58.561139 1165 raft_consensus.cc:357] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.561331 1165 raft_consensus.cc:383] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.561383 1165 raft_consensus.cc:738] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.561553 1165 consensus_queue.cc:260] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.561960 1165 ts_tablet_manager.cc:1428] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:52:58.561373 1176 raft_consensus.cc:357] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.562793 1176 raft_consensus.cc:383] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.562848 1176 raft_consensus.cc:738] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.563282 1176 consensus_queue.cc:260] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.563390 1165 tablet_bootstrap.cc:492] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:58.564458 1176 ts_tablet_manager.cc:1428] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:52:58.564706 1176 tablet_bootstrap.cc:492] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.565238 1158 tablet_bootstrap.cc:492] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.565390 1158 ts_tablet_manager.cc:1397] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20250109 20:52:58.565995 1158 raft_consensus.cc:357] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.566218 1158 raft_consensus.cc:383] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.566311 1158 raft_consensus.cc:738] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.566583 1158 consensus_queue.cc:260] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.567579 1158 ts_tablet_manager.cc:1428] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:58.568076 1158 tablet_bootstrap.cc:492] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.569514 1158 tablet_bootstrap.cc:654] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.569833 886 tablet_service.cc:1514] Processing DeleteTablet for tablet ca2b022ad9274c0e9b67560389d8dbed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.570104 1160 tablet_replica.cc:331] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:58.570252 1160 raft_consensus.cc:2238] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.570355 1160 raft_consensus.cc:2267] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.571023 1160 ts_tablet_manager.cc:1905] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.571842 1165 tablet_bootstrap.cc:654] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.576435 1160 ts_tablet_manager.cc:1918] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.576519 1160 log.cc:1198] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ca2b022ad9274c0e9b67560389d8dbed
I20250109 20:52:58.578159 1160 ts_tablet_manager.cc:1939] T ca2b022ad9274c0e9b67560389d8dbed P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:58.579049 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ca2b022ad9274c0e9b67560389d8dbed (table default.Table_c3d79b7bc6ba49c1ac4e56921b055cbb [id=5d0cc83d6e3e4da4845ec153a35cd890]) successfully deleted
I20250109 20:52:58.582139 2129 raft_consensus.cc:491] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.582252 2129 raft_consensus.cc:513] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.582865 2129 leader_election.cc:290] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.583174 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0d889cb1114edc8b799b41b23a54a6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:58.583269 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0d889cb1114edc8b799b41b23a54a6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.583715 990 leader_election.cc:343] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:58.583889 989 leader_election.cc:343] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.583995 989 leader_election.cc:304] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.584303 2129 raft_consensus.cc:2743] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.589919 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7768e94e2bff455eb490b7cd4e6f1fbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.590556 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7768e94e2bff455eb490b7cd4e6f1fbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.592510 1165 tablet_bootstrap.cc:492] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:58.592772 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4a5f5d985f414ece9c3a1e4962abe0b7" } new_table_name: "default.Table_6dbb42e1142c41af8ac284b91d5fe4a1" modify_external_catalogs: true
I20250109 20:52:58.593206 1176 tablet_bootstrap.cc:654] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.593217 1165 ts_tablet_manager.cc:1397] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.003s
I20250109 20:52:58.594548 1165 raft_consensus.cc:357] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.594749 1165 raft_consensus.cc:383] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.594803 1165 raft_consensus.cc:738] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.595031 1165 consensus_queue.cc:260] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.596055 1165 ts_tablet_manager.cc:1428] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:58.596339 1165 tablet_bootstrap.cc:492] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:58.596448 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f2388e84f29e48a9b66925c5271adcdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.596911 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2388e84f29e48a9b66925c5271adcdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.599301 1165 tablet_bootstrap.cc:654] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.601114 1158 tablet_bootstrap.cc:492] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.601245 1158 ts_tablet_manager.cc:1397] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.009s
I20250109 20:52:58.601822 1158 raft_consensus.cc:357] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.601984 1158 raft_consensus.cc:383] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.602031 1158 raft_consensus.cc:738] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.602299 1158 consensus_queue.cc:260] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.602836 1176 tablet_bootstrap.cc:492] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.602996 1176 ts_tablet_manager.cc:1397] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.001s
I20250109 20:52:58.603792 1176 raft_consensus.cc:357] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.604764 1158 ts_tablet_manager.cc:1428] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:58.605213 1158 tablet_bootstrap.cc:492] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.604007 1176 raft_consensus.cc:383] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.606520 1176 raft_consensus.cc:738] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.606775 1176 consensus_queue.cc:260] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.607350 1176 ts_tablet_manager.cc:1428] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:58.607637 1176 tablet_bootstrap.cc:492] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.610505 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1bc42ce5b4ac4d4283cdd4117d6fc80a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.611119 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bc42ce5b4ac4d4283cdd4117d6fc80a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.612632 1176 tablet_bootstrap.cc:654] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.613003 886 tablet_service.cc:1514] Processing DeleteTablet for tablet a8da9781ab32429297d0c875c1971501 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.613250 1158 tablet_bootstrap.cc:654] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.615222 1160 tablet_replica.cc:331] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:58.615459 1160 raft_consensus.cc:2238] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.615620 1160 raft_consensus.cc:2267] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.615728 911 raft_consensus.cc:3054] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:58.616369 1160 ts_tablet_manager.cc:1905] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:58.616987 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b10e70b6b88f438392a63a46123bdad0 consensus state has no leader
I20250109 20:52:58.618496 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6ba9903051db4a88adae464061f959d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.618975 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ba9903051db4a88adae464061f959d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.622088 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6c649042616d4c449929c794c0ba36cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.622535 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c649042616d4c449929c794c0ba36cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.623003 1160 ts_tablet_manager.cc:1918] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.623149 1160 log.cc:1198] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/a8da9781ab32429297d0c875c1971501
I20250109 20:52:58.624331 1160 ts_tablet_manager.cc:1939] T a8da9781ab32429297d0c875c1971501 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:58.625684 1158 tablet_bootstrap.cc:492] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.625707 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet a8da9781ab32429297d0c875c1971501 (table default.Table_4c30a9425b844ee187a29bc12cd2704e [id=8fddac7b2a084c15b76824d25240859d]) successfully deleted
I20250109 20:52:58.625809 1158 ts_tablet_manager.cc:1397] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250109 20:52:58.626597 1165 tablet_bootstrap.cc:492] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:58.626508 1158 raft_consensus.cc:357] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.626672 1158 raft_consensus.cc:383] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.626734 1158 raft_consensus.cc:738] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.626856 1165 ts_tablet_manager.cc:1397] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.010s
I20250109 20:52:58.627063 1158 consensus_queue.cc:260] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.627233 1984 raft_consensus.cc:491] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.627324 1984 raft_consensus.cc:513] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.627640 1165 raft_consensus.cc:357] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.627840 1165 raft_consensus.cc:383] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.627856 1984 leader_election.cc:290] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.627954 1165 raft_consensus.cc:738] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:52:58.628803 713 leader_election.cc:343] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8a5a6dbb823e4bd3afeb157b3c5771ba
I20250109 20:52:58.628434 1165 consensus_queue.cc:260] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.628852 1158 ts_tablet_manager.cc:1428] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:58.629467 1158 tablet_bootstrap.cc:492] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.631074 1165 ts_tablet_manager.cc:1428] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:58.631351 1165 tablet_bootstrap.cc:492] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:58.631780 1158 tablet_bootstrap.cc:654] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.633194 1165 tablet_bootstrap.cc:654] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.635324 1984 raft_consensus.cc:491] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.635445 1984 raft_consensus.cc:513] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.636011 1984 leader_election.cc:290] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.636521 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5ea5d588834f1b8b2b057ec324aadc" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.636759 713 leader_election.cc:343] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: ed5ea5d588834f1b8b2b057ec324aadc
W20250109 20:52:58.636938 712 leader_election.cc:343] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.637035 712 leader_election.cc:304] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.637346 1984 raft_consensus.cc:2743] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.639016 1165 tablet_bootstrap.cc:492] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:58.639166 1165 ts_tablet_manager.cc:1397] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20250109 20:52:58.639786 1165 raft_consensus.cc:357] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.640029 1165 raft_consensus.cc:383] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.640120 1165 raft_consensus.cc:738] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.640369 1165 consensus_queue.cc:260] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.641964 1165 ts_tablet_manager.cc:1428] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:58.642253 1165 tablet_bootstrap.cc:492] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:58.642910 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a5a6dbb823e4bd3afeb157b3c5771ba" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.643378 712 leader_election.cc:343] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.643496 712 leader_election.cc:304] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.643611 1176 tablet_bootstrap.cc:492] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.643719 1176 ts_tablet_manager.cc:1397] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.001s
I20250109 20:52:58.643800 1984 raft_consensus.cc:2743] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.644354 1176 raft_consensus.cc:357] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.644517 1176 raft_consensus.cc:383] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.644570 1176 raft_consensus.cc:738] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.644855 1176 consensus_queue.cc:260] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.645443 1176 ts_tablet_manager.cc:1428] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:58.646044 1176 tablet_bootstrap.cc:492] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.646032 1165 tablet_bootstrap.cc:654] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.647693 1176 tablet_bootstrap.cc:654] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.652356 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6dc6acbd34db412caf3281183b3c0f74" } modify_external_catalogs: true
I20250109 20:52:58.652545 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6dc6acbd34db412caf3281183b3c0f74" } modify_external_catalogs: true
I20250109 20:52:58.652935 1984 raft_consensus.cc:491] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.653072 1984 raft_consensus.cc:513] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.653753 1984 leader_election.cc:290] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:52:58.655296 713 leader_election.cc:343] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 91dbccae0af24a629600a92e7eadfee7
I20250109 20:52:58.656776 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_29736471ff5740c6909a38f7df420f22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.657620 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29736471ff5740c6909a38f7df420f22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.658627 1176 tablet_bootstrap.cc:492] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.658751 1176 ts_tablet_manager.cc:1397] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20250109 20:52:58.659404 1176 raft_consensus.cc:357] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.659560 1176 raft_consensus.cc:383] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.659605 1176 raft_consensus.cc:738] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.659813 1176 consensus_queue.cc:260] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.660579 1176 ts_tablet_manager.cc:1428] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:58.661367 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91dbccae0af24a629600a92e7eadfee7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:58.661645 1165 tablet_bootstrap.cc:492] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:58.661756 1165 ts_tablet_manager.cc:1397] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.009s
W20250109 20:52:58.662635 712 leader_election.cc:343] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.662710 1176 tablet_bootstrap.cc:492] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.662739 712 leader_election.cc:304] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.662464 1165 raft_consensus.cc:357] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.663087 1165 raft_consensus.cc:383] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.663085 1678 raft_consensus.cc:2743] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.663143 1165 raft_consensus.cc:738] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.663331 1165 consensus_queue.cc:260] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.665745 1165 ts_tablet_manager.cc:1428] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:52:58.666060 1165 tablet_bootstrap.cc:492] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:58.667219 1176 tablet_bootstrap.cc:654] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.668113 1165 tablet_bootstrap.cc:654] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.674320 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a455033b674e5a94fc37d0275eb313 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.674396 1176 tablet_bootstrap.cc:492] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.674499 1176 ts_tablet_manager.cc:1397] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20250109 20:52:58.674154 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fe13baff8c5840dea64e9e438c5e2e49" } new_table_name: "default.Table_396b994bc33a452daf214a5a46ad5650" modify_external_catalogs: true
I20250109 20:52:58.674605 1160 tablet_replica.cc:331] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:58.674808 1160 raft_consensus.cc:2238] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.674937 1160 raft_consensus.cc:2267] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.675216 1176 raft_consensus.cc:357] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.675380 1176 raft_consensus.cc:383] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.675431 1176 raft_consensus.cc:738] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.677409 1176 consensus_queue.cc:260] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.677955 1176 ts_tablet_manager.cc:1428] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:52:58.678550 1176 tablet_bootstrap.cc:492] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.679261 886 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe8a9e5b5fe406b988126cd1859eabc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.679805 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3]). Expected version 2 got 0
I20250109 20:52:58.679891 1160 ts_tablet_manager.cc:1905] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.679915 1176 tablet_bootstrap.cc:654] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.680449 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ff19ec0caa6049ddbedbf619756c83bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.680924 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff19ec0caa6049ddbedbf619756c83bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.685006 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ed658ec6109f45998eb6e916f76e8a79" } new_table_name: "default.Table_e3593cefd5e948bfb6e25e78caaff8de" modify_external_catalogs: true
I20250109 20:52:58.685894 1158 tablet_bootstrap.cc:492] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.686034 1158 ts_tablet_manager.cc:1397] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.001s
I20250109 20:52:58.686074 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e56f30e6f3450a8426cd40db89e009 with delete_type TABLET_DATA_DELETED (replaced by tablet eaea4f239cce41a997496a1c94810ac9) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.686914 1158 raft_consensus.cc:357] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.687120 1158 raft_consensus.cc:383] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.687171 1158 raft_consensus.cc:738] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.687394 1158 consensus_queue.cc:260] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.687943 1158 ts_tablet_manager.cc:1428] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:52:58.688234 1158 tablet_bootstrap.cc:492] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.689320 2129 raft_consensus.cc:491] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.689421 2129 raft_consensus.cc:513] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.689952 2129 leader_election.cc:290] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:52:58.690685 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 57e56f30e6f3450a8426cd40db89e009 because the tablet was not found. No further retry: Not found: Tablet not found: 57e56f30e6f3450a8426cd40db89e009
I20250109 20:52:58.691217 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fb1017fe6c4f5abd640ad1f6214232" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:58.691654 990 leader_election.cc:343] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.691798 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fb1017fe6c4f5abd640ad1f6214232" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.692324 989 leader_election.cc:343] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.692428 989 leader_election.cc:304] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.692792 1158 tablet_bootstrap.cc:654] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.692967 2129 raft_consensus.cc:2743] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.693738 886 tablet_service.cc:1467] Processing CreateTablet for tablet 2603d752cb1549929ec1da91501ad57a (DEFAULT_TABLE table=default.Table_fe13baff8c5840dea64e9e438c5e2e49 [id=32ced70c28af4564bd8600a0591d84b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.694115 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2603d752cb1549929ec1da91501ad57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.694347 1160 ts_tablet_manager.cc:1918] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.694497 1160 log.cc:1198] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/72a455033b674e5a94fc37d0275eb313
I20250109 20:52:58.694926 1160 ts_tablet_manager.cc:1939] T 72a455033b674e5a94fc37d0275eb313 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:58.695959 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 72a455033b674e5a94fc37d0275eb313 (table default.Table_e6d0cb94669f451389e1f2b2581c63b6 [id=a9b4def3cf2e4d34a0b70bb6b34881bb]) successfully deleted
I20250109 20:52:58.696305 1160 tablet_replica.cc:331] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:58.696503 1160 raft_consensus.cc:2238] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.696681 1160 raft_consensus.cc:2267] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.697501 1160 ts_tablet_manager.cc:1905] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.698923 1176 tablet_bootstrap.cc:492] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.699090 1176 ts_tablet_manager.cc:1397] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.002s
I20250109 20:52:58.699162 1158 tablet_bootstrap.cc:492] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.699267 1158 ts_tablet_manager.cc:1397] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20250109 20:52:58.699913 1176 raft_consensus.cc:357] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.699916 1158 raft_consensus.cc:357] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.700105 1158 raft_consensus.cc:383] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.700199 1176 raft_consensus.cc:383] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.700165 1158 raft_consensus.cc:738] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.700326 1176 raft_consensus.cc:738] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.700516 1158 consensus_queue.cc:260] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.700572 1176 consensus_queue.cc:260] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.703703 1158 ts_tablet_manager.cc:1428] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250109 20:52:58.704169 1158 tablet_bootstrap.cc:492] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.705608 1160 ts_tablet_manager.cc:1918] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.707541 1176 ts_tablet_manager.cc:1428] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250109 20:52:58.707902 1176 tablet_bootstrap.cc:492] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.708649 1160 log.cc:1198] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/fbe8a9e5b5fe406b988126cd1859eabc
I20250109 20:52:58.709000 1160 ts_tablet_manager.cc:1939] T fbe8a9e5b5fe406b988126cd1859eabc P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:58.709208 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1c422468931d4ec18c7d8cf9c392d3a6
I20250109 20:52:58.709682 450 catalog_manager.cc:6596] ReplaceTablet: tablet 1c422468931d4ec18c7d8cf9c392d3a6 deleted and replaced by tablet 52b555c0449e4ed280c9116b151b3b7a
I20250109 20:52:58.710263 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c422468931d4ec18c7d8cf9c392d3a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 52b555c0449e4ed280c9116b151b3b7a) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.710613 1158 tablet_bootstrap.cc:654] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:58.710834 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1c422468931d4ec18c7d8cf9c392d3a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c422468931d4ec18c7d8cf9c392d3a6 already in progress: creating tablet
I20250109 20:52:58.711117 1176 tablet_bootstrap.cc:654] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.712694 1158 tablet_bootstrap.cc:492] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:58.712836 1158 ts_tablet_manager.cc:1397] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20250109 20:52:58.713074 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c422468931d4ec18c7d8cf9c392d3a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 52b555c0449e4ed280c9116b151b3b7a) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:58.713627 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1c422468931d4ec18c7d8cf9c392d3a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c422468931d4ec18c7d8cf9c392d3a6 already in progress: creating tablet
I20250109 20:52:58.713910 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet fbe8a9e5b5fe406b988126cd1859eabc (table default.Table_d4b677aa33534a4d8261f52878181252 [id=2f89d33acf544985bafb538d400c0d11]) successfully deleted
I20250109 20:52:58.714115 1158 raft_consensus.cc:357] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.714277 1158 raft_consensus.cc:383] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.714319 1158 raft_consensus.cc:738] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.715292 1158 consensus_queue.cc:260] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.716181 1158 ts_tablet_manager.cc:1428] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:52:58.716409 1158 tablet_bootstrap.cc:492] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:58.717955 1158 tablet_bootstrap.cc:654] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.724049 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c422468931d4ec18c7d8cf9c392d3a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 52b555c0449e4ed280c9116b151b3b7a) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:58.724810 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1c422468931d4ec18c7d8cf9c392d3a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c422468931d4ec18c7d8cf9c392d3a6 already in progress: creating tablet
I20250109 20:52:58.727795 1176 tablet_bootstrap.cc:492] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:58.728004 1176 ts_tablet_manager.cc:1397] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20250109 20:52:58.729018 1176 raft_consensus.cc:357] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.729334 1176 raft_consensus.cc:383] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:58.729419 1176 raft_consensus.cc:738] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:58.729714 1176 consensus_queue.cc:260] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.730860 740 tablet_service.cc:1467] Processing CreateTablet for tablet 2603d752cb1549929ec1da91501ad57a (DEFAULT_TABLE table=default.Table_fe13baff8c5840dea64e9e438c5e2e49 [id=32ced70c28af4564bd8600a0591d84b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.731314 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2603d752cb1549929ec1da91501ad57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.732519 1176 ts_tablet_manager.cc:1428] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:58.733062 1176 tablet_bootstrap.cc:492] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:58.735606 1176 tablet_bootstrap.cc:654] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:58.737915 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 2603d752cb1549929ec1da91501ad57a (DEFAULT_TABLE table=default.Table_fe13baff8c5840dea64e9e438c5e2e49 [id=32ced70c28af4564bd8600a0591d84b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.738324 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2603d752cb1549929ec1da91501ad57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.746881 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3267402bd54b9bb1438a1e15424b3a (DEFAULT_TABLE table=default.Table_beeef07faf3a416aa68ffed7aab3a7b7 [id=15ba6e8a354b45069b13d88c367ceb65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.747189 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3267402bd54b9bb1438a1e15424b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.748320 886 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3267402bd54b9bb1438a1e15424b3a (DEFAULT_TABLE table=default.Table_beeef07faf3a416aa68ffed7aab3a7b7 [id=15ba6e8a354b45069b13d88c367ceb65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.748653 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3267402bd54b9bb1438a1e15424b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.749788 2129 raft_consensus.cc:491] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.749903 2129 raft_consensus.cc:513] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.750478 2129 leader_election.cc:290] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.751050 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16a1d9a5b374b8dbf725eb32ffadb9e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:58.751724 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16a1d9a5b374b8dbf725eb32ffadb9e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.752130 989 leader_election.cc:343] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.752279 2125 tablet.cc:1721] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250109 20:52:58.752535 990 leader_election.cc:343] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.752645 990 leader_election.cc:304] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.753472 2129 raft_consensus.cc:2743] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.753703 2130 tablet.cc:1721] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250109 20:52:58.754562 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2603d752cb1549929ec1da91501ad57a consensus state has no leader
I20250109 20:52:58.755151 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7fb50e2939fb4fe39c63669d58697188"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.755692 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fb50e2939fb4fe39c63669d58697188 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.762463 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 88cd7f9808d547ca80a76e4b9568cd0a
I20250109 20:52:58.763314 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 88cd7f9808d547ca80a76e4b9568cd0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.763223 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_696e8a1e0e3145bab03b3d71484d37cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.763840 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_696e8a1e0e3145bab03b3d71484d37cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:58.763885 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 88cd7f9808d547ca80a76e4b9568cd0a because the tablet was not found. No further retry: Not found: Tablet not found: 88cd7f9808d547ca80a76e4b9568cd0a
I20250109 20:52:58.765822 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 88cd7f9808d547ca80a76e4b9568cd0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.766562 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9558dfe563d94482bf496e19b8a80edc
I20250109 20:52:58.767134 451 catalog_manager.cc:6596] ReplaceTablet: tablet 9558dfe563d94482bf496e19b8a80edc deleted and replaced by tablet a8ad256c6bf64a699ede8ff3cee5dac1
I20250109 20:52:58.767334 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9558dfe563d94482bf496e19b8a80edc with delete_type TABLET_DATA_DELETED (replaced by tablet a8ad256c6bf64a699ede8ff3cee5dac1) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.767481 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5af25870e6cc4806873d8324713da6c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.767858 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9558dfe563d94482bf496e19b8a80edc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9558dfe563d94482bf496e19b8a80edc already in progress: creating tablet
W20250109 20:52:58.767925 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5af25870e6cc4806873d8324713da6c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.769428 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e6dba2d7cf654d449a1c9ff85ac1556f
I20250109 20:52:58.769969 755 tablet_service.cc:1514] Processing DeleteTablet for tablet e6dba2d7cf654d449a1c9ff85ac1556f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.770112 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_24291889d1144434be6778475dccf4a2" } new_table_name: "default.Table_24614ad1648f41d9bc4bb77eb2b170ae" modify_external_catalogs: true
W20250109 20:52:58.770566 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e6dba2d7cf654d449a1c9ff85ac1556f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6dba2d7cf654d449a1c9ff85ac1556f already in progress: creating tablet
I20250109 20:52:58.770900 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 9558dfe563d94482bf496e19b8a80edc with delete_type TABLET_DATA_DELETED (replaced by tablet a8ad256c6bf64a699ede8ff3cee5dac1) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.771086 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet e6dba2d7cf654d449a1c9ff85ac1556f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.771224 1166 tablet_replica.cc:331] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.771373 1166 raft_consensus.cc:2238] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.771498 1166 raft_consensus.cc:2267] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.771854 892 tablet_service.cc:1514] Processing DeleteTablet for tablet e6dba2d7cf654d449a1c9ff85ac1556f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.771940 1166 ts_tablet_manager.cc:1905] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.772367 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 9558dfe563d94482bf496e19b8a80edc with delete_type TABLET_DATA_DELETED (replaced by tablet a8ad256c6bf64a699ede8ff3cee5dac1) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:58.772881 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 88cd7f9808d547ca80a76e4b9568cd0a because the tablet was not found. No further retry: Not found: Tablet not found: 88cd7f9808d547ca80a76e4b9568cd0a
W20250109 20:52:58.773459 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e6dba2d7cf654d449a1c9ff85ac1556f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6dba2d7cf654d449a1c9ff85ac1556f already in progress: creating tablet
W20250109 20:52:58.773697 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9558dfe563d94482bf496e19b8a80edc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9558dfe563d94482bf496e19b8a80edc already in progress: creating tablet
W20250109 20:52:58.777369 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f53cef0adbb47d4840e4c11a6d0c8ac consensus state has no leader
I20250109 20:52:58.777630 1166 ts_tablet_manager.cc:1918] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.777729 1166 log.cc:1198] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/9558dfe563d94482bf496e19b8a80edc
I20250109 20:52:58.777745 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_17e950d4133f4d2a8eeab7f7af0b10c2" } new_table_name: "default.Table_7db9b887bea4440291c4923dbbe9e58e" modify_external_catalogs: true
I20250109 20:52:58.778067 1166 ts_tablet_manager.cc:1939] T 9558dfe563d94482bf496e19b8a80edc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.779735 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 9558dfe563d94482bf496e19b8a80edc (table default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6]) successfully deleted
W20250109 20:52:58.780248 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e6dba2d7cf654d449a1c9ff85ac1556f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6dba2d7cf654d449a1c9ff85ac1556f already in progress: creating tablet
I20250109 20:52:58.785308 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1f4279f07e8b4e0a86928bdb9a67bbf6" } modify_external_catalogs: true
I20250109 20:52:58.785526 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1f4279f07e8b4e0a86928bdb9a67bbf6" } modify_external_catalogs: true
I20250109 20:52:58.788025 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]). Expected version 1 got 0
I20250109 20:52:58.788223 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 14fb1017fe6c4f5abd640ad1f6214232 (table default.Table_6c787d10220c41dca63172c9b3bf70ae [id=26365b65a557409abc651578b4458fd4]). Expected version 1 got 0
I20250109 20:52:58.788427 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:52:58.788910 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7da1f3cafffe4f7ca39624b43831689e
I20250109 20:52:58.789664 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da1f3cafffe4f7ca39624b43831689e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.789970 899 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da1f3cafffe4f7ca39624b43831689e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:58.790547 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da1f3cafffe4f7ca39624b43831689e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:58.790628 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7da1f3cafffe4f7ca39624b43831689e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da1f3cafffe4f7ca39624b43831689e already in progress: creating tablet
W20250109 20:52:58.790699 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7da1f3cafffe4f7ca39624b43831689e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da1f3cafffe4f7ca39624b43831689e already in progress: creating tablet
I20250109 20:52:58.790748 1166 tablet_replica.cc:331] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.790937 1166 raft_consensus.cc:2238] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.791108 1166 raft_consensus.cc:2267] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.791905 1166 ts_tablet_manager.cc:1905] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.793308 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.793464 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.793849 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.794001 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e46f4312844b6eb29d6718f4800f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.794154 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet cd23012af5574f37944f9e26c07d05d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.794618 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.794792 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd758fa2fa4cebbdf7faf3856f1941) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.794946 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ca816507d0c047668a51f6e7ebce18cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce0a941f0c14501a947a5c7e2a844e8) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.798828 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 88cd7f9808d547ca80a76e4b9568cd0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:58.800742 1166 ts_tablet_manager.cc:1918] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.800881 1166 log.cc:1198] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/7da1f3cafffe4f7ca39624b43831689e
I20250109 20:52:58.801254 1166 ts_tablet_manager.cc:1939] T 7da1f3cafffe4f7ca39624b43831689e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.802275 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 7da1f3cafffe4f7ca39624b43831689e (table default.Table_4f4640c9222649fc957636279628a391 [id=cc1214f3eaf54bf0ba8167dbd8a4e634]) successfully deleted
I20250109 20:52:58.802579 1166 tablet_replica.cc:331] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.802771 1166 raft_consensus.cc:2238] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.802943 1166 raft_consensus.cc:2267] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.803738 1166 ts_tablet_manager.cc:1905] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.804350 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:52:58.814514 1166 ts_tablet_manager.cc:1918] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.821609 755 tablet_service.cc:1467] Processing CreateTablet for tablet 0e5820a9fe844227aba8ed848605dcc1 (DEFAULT_TABLE table=default.Table_1ce8dba031c24ab2bd13cbaa5d99c9fd [id=e8a1bd50583b4ac0b4b5fefff1e827d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.821934 1166 log.cc:1198] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/a0d09daa2eed41c3952fa21aa2cdd09e
I20250109 20:52:58.822003 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e5820a9fe844227aba8ed848605dcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.822829 1166 ts_tablet_manager.cc:1939] T a0d09daa2eed41c3952fa21aa2cdd09e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.824054 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet a0d09daa2eed41c3952fa21aa2cdd09e (table default.Table_14ad6cd05135486e87607f62e9dd62c4 [id=eccfc40a5a774588b25444c9d6f1507d]) successfully deleted
I20250109 20:52:58.825134 899 tablet_service.cc:1467] Processing CreateTablet for tablet 0e5820a9fe844227aba8ed848605dcc1 (DEFAULT_TABLE table=default.Table_1ce8dba031c24ab2bd13cbaa5d99c9fd [id=e8a1bd50583b4ac0b4b5fefff1e827d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.825647 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e5820a9fe844227aba8ed848605dcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.828285 1166 tablet_replica.cc:331] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.828503 1166 raft_consensus.cc:2238] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.828673 1166 raft_consensus.cc:2267] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.830027 1166 ts_tablet_manager.cc:1905] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.832728 1678 raft_consensus.cc:491] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.832914 1678 raft_consensus.cc:513] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.833612 1678 leader_election.cc:290] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.833930 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ed00f7954143c9ba9bf65d8a8a68d2" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.834441 712 leader_election.cc:343] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:58.834600 713 leader_election.cc:343] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: a1ed00f7954143c9ba9bf65d8a8a68d2
I20250109 20:52:58.834671 713 leader_election.cc:304] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.835062 1678 raft_consensus.cc:2743] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.842473 446 master_service.cc:946] ReplaceTablet: received request to replace tablet b56d2c0706c84195acceb97ef4985c14 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:58.844431 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3267402bd54b9bb1438a1e15424b3a (DEFAULT_TABLE table=default.Table_beeef07faf3a416aa68ffed7aab3a7b7 [id=15ba6e8a354b45069b13d88c367ceb65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.845021 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3267402bd54b9bb1438a1e15424b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.846541 2129 raft_consensus.cc:491] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.846683 2129 raft_consensus.cc:513] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.847411 2129 leader_election.cc:290] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.848044 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd23012af5574f37944f9e26c07d05d2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:58.848773 990 leader_election.cc:343] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.849480 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd23012af5574f37944f9e26c07d05d2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.850175 989 leader_election.cc:343] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.850291 989 leader_election.cc:304] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.850760 2129 raft_consensus.cc:2743] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.862020 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]). Expected version 2 got 0
I20250109 20:52:58.866431 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 0e5820a9fe844227aba8ed848605dcc1 (DEFAULT_TABLE table=default.Table_1ce8dba031c24ab2bd13cbaa5d99c9fd [id=e8a1bd50583b4ac0b4b5fefff1e827d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.866796 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e5820a9fe844227aba8ed848605dcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.867965 1166 ts_tablet_manager.cc:1918] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.868047 1166 log.cc:1198] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/e16a1d9a5b374b8dbf725eb32ffadb9e
I20250109 20:52:58.868072 755 tablet_service.cc:1467] Processing CreateTablet for tablet e3036ddd0d3f41f297c4e583ef6e5262 (DEFAULT_TABLE table=default.Table_23f5de34735e490a84de4f459a0258f7 [id=8998090fe3414a4ba1d9066653eb1a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.868505 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3036ddd0d3f41f297c4e583ef6e5262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.872598 899 tablet_service.cc:1467] Processing CreateTablet for tablet e3036ddd0d3f41f297c4e583ef6e5262 (DEFAULT_TABLE table=default.Table_23f5de34735e490a84de4f459a0258f7 [id=8998090fe3414a4ba1d9066653eb1a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.868335 1166 ts_tablet_manager.cc:1939] T e16a1d9a5b374b8dbf725eb32ffadb9e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.873176 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3036ddd0d3f41f297c4e583ef6e5262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.873917 1166 tablet_replica.cc:331] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.874167 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet e16a1d9a5b374b8dbf725eb32ffadb9e (table default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]) successfully deleted
I20250109 20:52:58.875859 1166 raft_consensus.cc:2238] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.879189 1166 raft_consensus.cc:2267] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.880360 1166 ts_tablet_manager.cc:1905] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.883898 2129 raft_consensus.cc:491] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.885799 2129 raft_consensus.cc:513] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.886413 2129 leader_election.cc:290] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.887002 911 raft_consensus.cc:3054] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:58.886991 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e46f4312844b6eb29d6718f4800f98" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:58.887696 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e46f4312844b6eb29d6718f4800f98" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.888176 989 leader_election.cc:343] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:58.888514 990 leader_election.cc:343] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.888605 990 leader_election.cc:304] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.888993 2129 raft_consensus.cc:2743] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.890349 1020 tablet_service.cc:1467] Processing CreateTablet for tablet e3036ddd0d3f41f297c4e583ef6e5262 (DEFAULT_TABLE table=default.Table_23f5de34735e490a84de4f459a0258f7 [id=8998090fe3414a4ba1d9066653eb1a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.890722 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3036ddd0d3f41f297c4e583ef6e5262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.893342 892 tablet_service.cc:1467] Processing CreateTablet for tablet a8c0f5dc3c794811b55040dcc094246c (DEFAULT_TABLE table=default.Table_cb6e0dd9d8ff4cde977360e8da20fd10 [id=7a3a8cd9aeb6470da428367c2095cc4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.893728 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c0f5dc3c794811b55040dcc094246c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.897255 2126 tablet.cc:1721] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:58.905694 740 tablet_service.cc:1467] Processing CreateTablet for tablet a8c0f5dc3c794811b55040dcc094246c (DEFAULT_TABLE table=default.Table_cb6e0dd9d8ff4cde977360e8da20fd10 [id=7a3a8cd9aeb6470da428367c2095cc4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.906093 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c0f5dc3c794811b55040dcc094246c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.913280 1166 ts_tablet_manager.cc:1918] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.913461 1166 log.cc:1198] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/bb0d889cb1114edc8b799b41b23a54a6
I20250109 20:52:58.914877 1020 tablet_service.cc:1467] Processing CreateTablet for tablet a8c0f5dc3c794811b55040dcc094246c (DEFAULT_TABLE table=default.Table_cb6e0dd9d8ff4cde977360e8da20fd10 [id=7a3a8cd9aeb6470da428367c2095cc4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.917382 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c0f5dc3c794811b55040dcc094246c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.917382 1166 ts_tablet_manager.cc:1939] T bb0d889cb1114edc8b799b41b23a54a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.918932 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet bb0d889cb1114edc8b799b41b23a54a6 (table default.Table_0bef1709ef9c47738d52c3d41d4f6819 [id=65098b4e94ce440494b2d5682132ba1a]) successfully deleted
I20250109 20:52:58.922461 911 raft_consensus.cc:3054] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:58.923249 1166 tablet_replica.cc:331] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.923408 1166 raft_consensus.cc:2238] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.923527 1166 raft_consensus.cc:2267] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.927455 892 tablet_service.cc:1467] Processing CreateTablet for tablet c6e4cfb5a07841828e5206d100b4f835 (DEFAULT_TABLE table=default.Table_0ef997a787824972852640e0fa2bfebb [id=96f454e79051412b98fa7863c2175096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.928211 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6e4cfb5a07841828e5206d100b4f835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.932508 1166 ts_tablet_manager.cc:1905] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.950696 1166 ts_tablet_manager.cc:1918] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:58.951128 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 733c7059c750463aa586d9b733f1d1e9 consensus state has no leader
I20250109 20:52:58.952344 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3]). Expected version 2 got 0
I20250109 20:52:58.952399 1166 log.cc:1198] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/04e46f4312844b6eb29d6718f4800f98
W20250109 20:52:58.952440 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d3a253685e34e80a26f338be66d31bb consensus state has no leader
I20250109 20:52:58.952697 1166 ts_tablet_manager.cc:1939] T 04e46f4312844b6eb29d6718f4800f98 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.952813 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_43ac00c1a42746499f265e3beafcc8c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:58.953130 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6e020eeb03e041ed977779a07de5bb80
W20250109 20:52:58.953390 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43ac00c1a42746499f265e3beafcc8c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.953877 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_27a065584631460fa3736a814a26fad6" } new_table_name: "default.Table_79fb13be9f0440c9991b67d641ab0b6a" modify_external_catalogs: true
I20250109 20:52:58.953974 1166 tablet_replica.cc:331] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.954162 1166 raft_consensus.cc:2238] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.954205 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 04e46f4312844b6eb29d6718f4800f98 (table default.Table_c4b9fe08755e4cb1b56978cbda1f2025 [id=4394b742727948f99337b3aa45e81de4]) successfully deleted
I20250109 20:52:58.954293 1166 raft_consensus.cc:2267] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.954449 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e020eeb03e041ed977779a07de5bb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.954721 1166 ts_tablet_manager.cc:1905] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.954780 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f9b7d3b2a65544bf894ec96e183e2d8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.955062 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6e020eeb03e041ed977779a07de5bb80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e020eeb03e041ed977779a07de5bb80 already in progress: creating tablet
W20250109 20:52:58.955240 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9b7d3b2a65544bf894ec96e183e2d8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.955865 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e020eeb03e041ed977779a07de5bb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:58.956576 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6e020eeb03e041ed977779a07de5bb80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e020eeb03e041ed977779a07de5bb80 already in progress: creating tablet
I20250109 20:52:58.956498 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_31e395d70eeb405c97b2221c23dd329a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.956930 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31e395d70eeb405c97b2221c23dd329a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.959405 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2ba81d736745fb9d56fd9a7c229153 (DEFAULT_TABLE table=default.Table_7dbdb0f9160444ca9abff301774ac41d [id=298a30a3008046b1bb63d18f4b21e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.959493 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c6e4cfb5a07841828e5206d100b4f835 (DEFAULT_TABLE table=default.Table_0ef997a787824972852640e0fa2bfebb [id=96f454e79051412b98fa7863c2175096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.959770 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2ba81d736745fb9d56fd9a7c229153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.959825 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6e4cfb5a07841828e5206d100b4f835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.961925 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 440e6af656094411b094c680c561dfc2
I20250109 20:52:58.962538 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 440e6af656094411b094c680c561dfc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:58.962590 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9ff1145eaa7f496b80139a83927b023a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:58.962993 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff1145eaa7f496b80139a83927b023a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.963220 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 440e6af656094411b094c680c561dfc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:58.963400 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 440e6af656094411b094c680c561dfc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 440e6af656094411b094c680c561dfc2 already in progress: creating tablet
I20250109 20:52:58.964268 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 440e6af656094411b094c680c561dfc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:58.964849 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 440e6af656094411b094c680c561dfc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 440e6af656094411b094c680c561dfc2 already in progress: creating tablet
I20250109 20:52:58.967759 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6c856ba75072493c881fa3bb24292947"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:58.969168 1678 raft_consensus.cc:491] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.969318 1678 raft_consensus.cc:513] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:58.969688 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c856ba75072493c881fa3bb24292947 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:58.970005 1678 leader_election.cc:290] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.970425 1166 ts_tablet_manager.cc:1918] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.970510 1166 log.cc:1198] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/cd23012af5574f37944f9e26c07d05d2
I20250109 20:52:58.970779 1166 ts_tablet_manager.cc:1939] T cd23012af5574f37944f9e26c07d05d2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.970834 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6619b3dce3ed4325a4871c3d40b42c9d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:58.971096 1055 raft_consensus.cc:2463] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:58.971715 713 leader_election.cc:304] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:58.971870 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet cd23012af5574f37944f9e26c07d05d2 (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:52:58.972131 1678 raft_consensus.cc:2798] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:58.972221 1678 raft_consensus.cc:491] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:58.972293 1678 raft_consensus.cc:3054] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:58.975309 1678 raft_consensus.cc:513] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.976487 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6619b3dce3ed4325a4871c3d40b42c9d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:58.977386 712 leader_election.cc:343] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.977938 1166 tablet_replica.cc:331] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.978166 1166 raft_consensus.cc:2238] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.978212 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6619b3dce3ed4325a4871c3d40b42c9d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:58.978309 1166 raft_consensus.cc:2267] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.978365 1055 raft_consensus.cc:3054] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:58.979562 1984 raft_consensus.cc:491] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.979687 1166 ts_tablet_manager.cc:1905] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.979662 1984 raft_consensus.cc:513] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.980203 1984 leader_election.cc:290] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.980731 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47726b58d6714cd1b46848a16a34ff37" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.981228 712 leader_election.cc:343] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:58.982357 713 leader_election.cc:343] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 47726b58d6714cd1b46848a16a34ff37
I20250109 20:52:58.982501 713 leader_election.cc:304] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.982954 1975 raft_consensus.cc:491] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.983135 1975 raft_consensus.cc:513] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.983788 1975 leader_election.cc:290] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.983968 1975 raft_consensus.cc:2743] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.984633 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ac8d15d48c49289712ce27768dc27b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.985284 712 leader_election.cc:343] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:58.987385 713 leader_election.cc:343] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 71ac8d15d48c49289712ce27768dc27b
I20250109 20:52:58.987519 713 leader_election.cc:304] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:58.987953 1166 ts_tablet_manager.cc:1918] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:58.987964 1984 raft_consensus.cc:2743] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.988334 1678 leader_election.cc:290] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:58.989235 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6619b3dce3ed4325a4871c3d40b42c9d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:58.989940 712 leader_election.cc:343] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.988768 2129 raft_consensus.cc:491] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:58.991372 1166 log.cc:1198] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/59ebeffe516c4c239ae2ee9040276e53
I20250109 20:52:58.991395 2129 raft_consensus.cc:513] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.991631 1166 ts_tablet_manager.cc:1939] T 59ebeffe516c4c239ae2ee9040276e53 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:58.991954 2129 leader_election.cc:290] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:58.992453 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 59ebeffe516c4c239ae2ee9040276e53 (table default.Table_887acb86017941c4b8636c1008d835d9 [id=facd2e83422c492aa4337beff9ceb548]) successfully deleted
I20250109 20:52:58.992522 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daac99da5d10475c9de263bef27baeb7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:58.993132 990 leader_election.cc:343] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.993698 1166 tablet_replica.cc:331] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:58.993855 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daac99da5d10475c9de263bef27baeb7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:58.994159 740 tablet_service.cc:1467] Processing CreateTablet for tablet c6e4cfb5a07841828e5206d100b4f835 (DEFAULT_TABLE table=default.Table_0ef997a787824972852640e0fa2bfebb [id=96f454e79051412b98fa7863c2175096]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:58.994274 989 leader_election.cc:343] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:58.994309 1166 raft_consensus.cc:2238] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:58.994364 989 leader_election.cc:304] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.988981 1055 raft_consensus.cc:2463] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:58.994741 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6e4cfb5a07841828e5206d100b4f835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:58.994958 2129 raft_consensus.cc:2743] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:58.995047 713 leader_election.cc:304] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:58.995190 1166 raft_consensus.cc:2267] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:58.995626 1678 raft_consensus.cc:2798] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:58.995921 1166 ts_tablet_manager.cc:1905] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:58.996805 1678 raft_consensus.cc:695] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:58.997330 1678 consensus_queue.cc:237] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:58.999254 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2ba81d736745fb9d56fd9a7c229153 (DEFAULT_TABLE table=default.Table_7dbdb0f9160444ca9abff301774ac41d [id=298a30a3008046b1bb63d18f4b21e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:58.999583 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2ba81d736745fb9d56fd9a7c229153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.004433 2129 raft_consensus.cc:491] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.004561 2129 raft_consensus.cc:513] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.005265 2129 leader_election.cc:290] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.005609 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.005970 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.006178 990 leader_election.cc:343] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.006603 989 leader_election.cc:343] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.006726 989 leader_election.cc:304] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.007135 2129 raft_consensus.cc:2743] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.008844 1166 ts_tablet_manager.cc:1918] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.009409 1166 log.cc:1198] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/72cc0bbbb5584f14ba4d6c62211a12c5
I20250109 20:52:59.009861 1166 ts_tablet_manager.cc:1939] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.010525 753 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc with delete_type TABLET_DATA_DELETED (replaced by tablet cd23012af5574f37944f9e26c07d05d2) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.010751 1157 tablet_replica.cc:331] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.011006 1157 raft_consensus.cc:2238] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.011163 1157 raft_consensus.cc:2267] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.011765 1157 ts_tablet_manager.cc:1905] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.011916 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 72cc0bbbb5584f14ba4d6c62211a12c5 (table default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]) successfully deleted
I20250109 20:52:59.012662 1166 tablet_replica.cc:331] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.013108 1166 raft_consensus.cc:2238] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.013520 1166 raft_consensus.cc:2267] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.014183 1166 ts_tablet_manager.cc:1905] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.019567 1157 ts_tablet_manager.cc:1918] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.019716 1157 log.cc:1198] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ed5ea5d588834f1b8b2b057ec324aadc
I20250109 20:52:59.020092 1157 ts_tablet_manager.cc:1939] T ed5ea5d588834f1b8b2b057ec324aadc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.021382 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ed5ea5d588834f1b8b2b057ec324aadc (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:52:59.024003 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2ba81d736745fb9d56fd9a7c229153 (DEFAULT_TABLE table=default.Table_7dbdb0f9160444ca9abff301774ac41d [id=298a30a3008046b1bb63d18f4b21e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.024219 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 71ac8d15d48c49289712ce27768dc27b with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.024398 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f2ba81d736745fb9d56fd9a7c229153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.024489 1157 tablet_replica.cc:331] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.024757 1157 raft_consensus.cc:2238] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.024974 1157 raft_consensus.cc:2267] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.025708 1157 ts_tablet_manager.cc:1905] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.027334 1984 raft_consensus.cc:491] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.027451 1984 raft_consensus.cc:513] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.028004 1984 leader_election.cc:290] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:59.028513 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e1e8754625463ea42a92ef65a89214" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.029083 712 leader_election.cc:343] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.029882 713 leader_election.cc:343] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 59e1e8754625463ea42a92ef65a89214
I20250109 20:52:59.030025 713 leader_election.cc:304] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:59.030385 1984 raft_consensus.cc:2743] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.030655 1157 ts_tablet_manager.cc:1918] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.030741 1157 log.cc:1198] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/71ac8d15d48c49289712ce27768dc27b
I20250109 20:52:59.031034 1157 ts_tablet_manager.cc:1939] T 71ac8d15d48c49289712ce27768dc27b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.031142 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e8754625463ea42a92ef65a89214 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a95a31468b431fb4304cf6680016da) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.031868 1157 tablet_replica.cc:331] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.031956 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 71ac8d15d48c49289712ce27768dc27b (table default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]) successfully deleted
I20250109 20:52:59.032073 1157 raft_consensus.cc:2238] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.032209 1157 raft_consensus.cc:2267] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.033010 1176 tablet_bootstrap.cc:492] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.033128 1176 ts_tablet_manager.cc:1397] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.300s user 0.022s sys 0.044s
I20250109 20:52:59.033814 1176 raft_consensus.cc:357] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.034032 1176 raft_consensus.cc:383] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.034032 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dbccae0af24a629600a92e7eadfee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.034075 1176 raft_consensus.cc:738] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.034271 1176 consensus_queue.cc:260] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.037961 1176 ts_tablet_manager.cc:1428] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:59.039361 1166 ts_tablet_manager.cc:1918] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.040015 1166 log.cc:1198] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ca816507d0c047668a51f6e7ebce18cb
I20250109 20:52:59.040391 1166 ts_tablet_manager.cc:1939] T ca816507d0c047668a51f6e7ebce18cb P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.041477 1176 tablet_bootstrap.cc:492] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.041630 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ca816507d0c047668a51f6e7ebce18cb (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]) successfully deleted
W20250109 20:52:59.042876 441 catalog_manager.cc:4839] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 88cd7f9808d547ca80a76e4b9568cd0a because the tablet was not found. No further retry: Not found: Tablet not found: 88cd7f9808d547ca80a76e4b9568cd0a
I20250109 20:52:59.043401 1176 tablet_bootstrap.cc:654] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:59.043907 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 440e6af656094411b094c680c561dfc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 440e6af656094411b094c680c561dfc2 already in progress: creating tablet
I20250109 20:52:59.045256 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.045805 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.046093 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 47726b58d6714cd1b46848a16a34ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.046281 1984 raft_consensus.cc:491] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.046481 1984 raft_consensus.cc:513] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.047679 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c3aa20539740cba4e5d6e852ad2909" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.050109 1176 tablet_bootstrap.cc:492] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.050254 1176 ts_tablet_manager.cc:1397] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:52:59.051017 1176 raft_consensus.cc:357] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.051225 1176 raft_consensus.cc:383] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.051276 1176 raft_consensus.cc:738] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.052354 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e020eeb03e041ed977779a07de5bb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.054040 1176 consensus_queue.cc:260] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.052598 1020 tablet_service.cc:1467] Processing CreateTablet for tablet e82cda1841c24bb2a14c127b1f40ad26 (DEFAULT_TABLE table=default.Table_a2dba4139826415bae8547cb15dd9863 [id=d18c1cee980b4e13a8930d7eb2361ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.054620 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82cda1841c24bb2a14c127b1f40ad26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.055171 1176 ts_tablet_manager.cc:1428] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:59.055187 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b56d2c0706c84195acceb97ef4985c14
I20250109 20:52:59.055636 446 catalog_manager.cc:6596] ReplaceTablet: tablet b56d2c0706c84195acceb97ef4985c14 deleted and replaced by tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2
I20250109 20:52:59.056000 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c075a9e071d84c4ca107998bcdc8af22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.056470 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c075a9e071d84c4ca107998bcdc8af22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.057137 1176 tablet_bootstrap.cc:492] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.057300 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_54a6148851094ffc9fb3923fb869a9e1" } new_table_name: "default.Table_0a1e37339bb647d48bcdabdc77edc400" modify_external_catalogs: true
I20250109 20:52:59.059576 739 tablet_service.cc:1467] Processing CreateTablet for tablet e82cda1841c24bb2a14c127b1f40ad26 (DEFAULT_TABLE table=default.Table_a2dba4139826415bae8547cb15dd9863 [id=d18c1cee980b4e13a8930d7eb2361ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.059952 1984 leader_election.cc:290] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:59.060081 890 tablet_service.cc:1467] Processing CreateTablet for tablet e233caae7679488d9b61f34e5f57b4d3 (DEFAULT_TABLE table=default.Table_aef4bd2a519c4d27964b01f6449b516b [id=a231ba433af748afa4c0fca588715b59]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:59.060132 712 leader_election.cc:343] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.060273 713 leader_election.cc:343] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 06c3aa20539740cba4e5d6e852ad2909
I20250109 20:52:59.060410 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e233caae7679488d9b61f34e5f57b4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.060611 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2ad301c11943b69b59be18d6775513 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b1822cd5744b5197d8141fc66539a0) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.060863 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82cda1841c24bb2a14c127b1f40ad26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.061883 1176 tablet_bootstrap.cc:654] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:59.063875 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6e020eeb03e041ed977779a07de5bb80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e020eeb03e041ed977779a07de5bb80 already in progress: creating tablet
I20250109 20:52:59.064177 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet b56d2c0706c84195acceb97ef4985c14 with delete_type TABLET_DATA_DELETED (replaced by tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:59.064786 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b56d2c0706c84195acceb97ef4985c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b56d2c0706c84195acceb97ef4985c14 already in progress: creating tablet
I20250109 20:52:59.065601 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3155ce67d2a64a1c8ebd1a308b8e33c7 (DEFAULT_TABLE table=default.Table_2fb311daf94d4ba582536280f88fd7b0 [id=0f2ad7100097472ca83e674594cf29de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.065793 1176 tablet_bootstrap.cc:492] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.065917 1176 ts_tablet_manager.cc:1397] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:52:59.065977 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3155ce67d2a64a1c8ebd1a308b8e33c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.065984 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3155ce67d2a64a1c8ebd1a308b8e33c7 (DEFAULT_TABLE table=default.Table_2fb311daf94d4ba582536280f88fd7b0 [id=0f2ad7100097472ca83e674594cf29de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.066329 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3155ce67d2a64a1c8ebd1a308b8e33c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.066494 1176 raft_consensus.cc:357] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.066635 1176 raft_consensus.cc:383] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.066673 1176 raft_consensus.cc:738] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.069157 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 3155ce67d2a64a1c8ebd1a308b8e33c7 (DEFAULT_TABLE table=default.Table_2fb311daf94d4ba582536280f88fd7b0 [id=0f2ad7100097472ca83e674594cf29de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.070900 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f3adc0bbe9994d269df1e03dd5541e33" } modify_external_catalogs: true
I20250109 20:52:59.071121 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3155ce67d2a64a1c8ebd1a308b8e33c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.071074 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f3adc0bbe9994d269df1e03dd5541e33" } modify_external_catalogs: true
I20250109 20:52:59.071588 1176 consensus_queue.cc:260] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.072650 1176 ts_tablet_manager.cc:1428] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:52:59.073186 1176 tablet_bootstrap.cc:492] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.074801 713 leader_election.cc:304] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:59.075222 1176 tablet_bootstrap.cc:654] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.075856 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_2fdd49df493a4c26bcdea27254701757 [id=2ab3ba8731124704889904608a2f0cd3] alter complete (version 2)
I20250109 20:52:59.076820 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b56d2c0706c84195acceb97ef4985c14 with delete_type TABLET_DATA_DELETED (replaced by tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:59.078377 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b56d2c0706c84195acceb97ef4985c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b56d2c0706c84195acceb97ef4985c14 already in progress: creating tablet
I20250109 20:52:59.080545 1984 raft_consensus.cc:2743] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.080821 899 tablet_service.cc:1467] Processing CreateTablet for tablet e82cda1841c24bb2a14c127b1f40ad26 (DEFAULT_TABLE table=default.Table_a2dba4139826415bae8547cb15dd9863 [id=d18c1cee980b4e13a8930d7eb2361ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.080878 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b56d2c0706c84195acceb97ef4985c14 with delete_type TABLET_DATA_DELETED (replaced by tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.081097 1157 ts_tablet_manager.cc:1905] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.081162 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82cda1841c24bb2a14c127b1f40ad26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.084110 1176 tablet_bootstrap.cc:492] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.084229 1176 ts_tablet_manager.cc:1397] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20250109 20:52:59.086158 1176 raft_consensus.cc:357] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.086344 1176 raft_consensus.cc:383] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.086390 1176 raft_consensus.cc:738] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.086561 753 tablet_service.cc:1467] Processing CreateTablet for tablet e233caae7679488d9b61f34e5f57b4d3 (DEFAULT_TABLE table=default.Table_aef4bd2a519c4d27964b01f6449b516b [id=a231ba433af748afa4c0fca588715b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.087087 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e233caae7679488d9b61f34e5f57b4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.088506 1176 consensus_queue.cc:260] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.092890 1176 ts_tablet_manager.cc:1428] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250109 20:52:59.093124 1176 tablet_bootstrap.cc:492] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.094843 1157 ts_tablet_manager.cc:1918] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.094977 1157 log.cc:1198] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/59e1e8754625463ea42a92ef65a89214
I20250109 20:52:59.095293 1157 ts_tablet_manager.cc:1939] T 59e1e8754625463ea42a92ef65a89214 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.095352 1176 tablet_bootstrap.cc:654] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.100032 1176 tablet_bootstrap.cc:492] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.100174 1176 ts_tablet_manager.cc:1397] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250109 20:52:59.100847 1176 raft_consensus.cc:357] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.101022 1176 raft_consensus.cc:383] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.101068 1176 raft_consensus.cc:738] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.106350 1176 consensus_queue.cc:260] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.111073 1020 tablet_service.cc:1467] Processing CreateTablet for tablet e233caae7679488d9b61f34e5f57b4d3 (DEFAULT_TABLE table=default.Table_aef4bd2a519c4d27964b01f6449b516b [id=a231ba433af748afa4c0fca588715b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.111413 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e233caae7679488d9b61f34e5f57b4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.111670 889 tablet_service.cc:1467] Processing CreateTablet for tablet cc720ad33da3460ea54e197229f489e1 (DEFAULT_TABLE table=default.Table_3757f39963744852aeac877e634f8416 [id=a39134842a1f48d2bbf4dc6eac21eeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.111757 753 tablet_service.cc:1467] Processing CreateTablet for tablet cc720ad33da3460ea54e197229f489e1 (DEFAULT_TABLE table=default.Table_3757f39963744852aeac877e634f8416 [id=a39134842a1f48d2bbf4dc6eac21eeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.111996 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc720ad33da3460ea54e197229f489e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.112035 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc720ad33da3460ea54e197229f489e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.113291 1176 ts_tablet_manager.cc:1428] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250109 20:52:59.113682 1176 tablet_bootstrap.cc:492] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.115583 1176 tablet_bootstrap.cc:654] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.118368 1176 tablet_bootstrap.cc:492] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.118474 1176 ts_tablet_manager.cc:1397] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:59.119184 1176 raft_consensus.cc:357] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.119396 1176 raft_consensus.cc:383] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.119439 1176 raft_consensus.cc:738] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.119647 1176 consensus_queue.cc:260] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.120769 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
I20250109 20:52:59.124178 1176 ts_tablet_manager.cc:1428] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:52:59.124406 1176 tablet_bootstrap.cc:492] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.127013 1176 tablet_bootstrap.cc:654] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.129163 450 catalog_manager.cc:5526] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:59.135599 1020 tablet_service.cc:1467] Processing CreateTablet for tablet cc720ad33da3460ea54e197229f489e1 (DEFAULT_TABLE table=default.Table_3757f39963744852aeac877e634f8416 [id=a39134842a1f48d2bbf4dc6eac21eeb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.135954 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc720ad33da3460ea54e197229f489e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.135993 1176 tablet_bootstrap.cc:492] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.136078 1176 ts_tablet_manager.cc:1397] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:52:59.136263 889 tablet_service.cc:1467] Processing CreateTablet for tablet b6bcf34f4788483783acb3d09accba37 (DEFAULT_TABLE table=default.Table_2972d7dcc5cd4f0e88717b5d94f99f4d [id=71df4d1acb7b45ef9f036afb5be2e915]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.136580 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6bcf34f4788483783acb3d09accba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.136693 1176 raft_consensus.cc:357] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.136868 1176 raft_consensus.cc:383] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.136907 1176 raft_consensus.cc:738] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.137068 1176 consensus_queue.cc:260] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.137691 886 tablet_service.cc:1467] Processing CreateTablet for tablet 420f755e81eb4ddd8b872bb7dcea4d13 (DEFAULT_TABLE table=default.Table_1c1f3e1543534694a3ccf10b8bf07284 [id=a7a6f3a773b6418f9acbedd58dc28bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.137980 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420f755e81eb4ddd8b872bb7dcea4d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.138222 747 tablet_service.cc:1467] Processing CreateTablet for tablet 420f755e81eb4ddd8b872bb7dcea4d13 (DEFAULT_TABLE table=default.Table_1c1f3e1543534694a3ccf10b8bf07284 [id=a7a6f3a773b6418f9acbedd58dc28bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.138525 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420f755e81eb4ddd8b872bb7dcea4d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.139582 1176 ts_tablet_manager.cc:1428] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250109 20:52:59.139787 1176 tablet_bootstrap.cc:492] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.140993 1029 tablet_service.cc:1467] Processing CreateTablet for tablet b6bcf34f4788483783acb3d09accba37 (DEFAULT_TABLE table=default.Table_2972d7dcc5cd4f0e88717b5d94f99f4d [id=71df4d1acb7b45ef9f036afb5be2e915]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.141333 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6bcf34f4788483783acb3d09accba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.141633 1176 tablet_bootstrap.cc:654] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.148501 1158 tablet_bootstrap.cc:492] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.148785 1158 ts_tablet_manager.cc:1397] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.432s user 0.076s sys 0.105s
I20250109 20:52:59.149785 1158 raft_consensus.cc:357] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.150086 1158 raft_consensus.cc:383] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.150182 1158 raft_consensus.cc:738] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.150421 1158 consensus_queue.cc:260] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.151109 1158 ts_tablet_manager.cc:1428] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:59.151422 1158 tablet_bootstrap.cc:492] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.153992 1158 tablet_bootstrap.cc:654] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.166128 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 420f755e81eb4ddd8b872bb7dcea4d13 (DEFAULT_TABLE table=default.Table_1c1f3e1543534694a3ccf10b8bf07284 [id=a7a6f3a773b6418f9acbedd58dc28bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.166489 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420f755e81eb4ddd8b872bb7dcea4d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.166723 747 tablet_service.cc:1467] Processing CreateTablet for tablet 801838c21f70424ba1b2d89585bda643 (DEFAULT_TABLE table=default.Table_261f03425c0e4e77873973fa3746a8c3 [id=ae99a336de5a4336bfaa8e4d2f8994e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.167124 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801838c21f70424ba1b2d89585bda643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.173084 1176 tablet_bootstrap.cc:492] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.173269 1176 ts_tablet_manager.cc:1397] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.006s
I20250109 20:52:59.173969 753 tablet_service.cc:1467] Processing CreateTablet for tablet b6bcf34f4788483783acb3d09accba37 (DEFAULT_TABLE table=default.Table_2972d7dcc5cd4f0e88717b5d94f99f4d [id=71df4d1acb7b45ef9f036afb5be2e915]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.174152 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 165f6304acc849569a8e90d135a8e2d0 (table default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320]). Expected version 1 got 0
I20250109 20:52:59.174351 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6bcf34f4788483783acb3d09accba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.174310 1176 raft_consensus.cc:357] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.174532 1176 raft_consensus.cc:383] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.174587 1176 raft_consensus.cc:738] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.174811 1176 consensus_queue.cc:260] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:59.175536 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee208bad76734a8fa1ada438bdbbe48c consensus state has no leader
I20250109 20:52:59.175575 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.175782 1166 tablet_replica.cc:331] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.175479 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.175840 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_90ff6b4b3aed4dcf8093176305e600e5" } modify_external_catalogs: true
I20250109 20:52:59.176016 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_90ff6b4b3aed4dcf8093176305e600e5" } modify_external_catalogs: true
I20250109 20:52:59.177335 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c696e743a5ae4b2ea41b48e15b437c31" } new_table_name: "default.Table_7147fbe7700d4c52a663ea8345c0a31d" modify_external_catalogs: true
I20250109 20:52:59.178205 1166 raft_consensus.cc:2238] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.178356 1166 raft_consensus.cc:2267] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.178656 1176 ts_tablet_manager.cc:1428] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:59.178754 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7d48fea31cc5403aaca9d51a191aa6d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.178897 1176 tablet_bootstrap.cc:492] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.179050 1166 ts_tablet_manager.cc:1905] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:59.179226 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d48fea31cc5403aaca9d51a191aa6d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.180377 1176 tablet_bootstrap.cc:654] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.180429 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_33b6f13c94484781be39523332e10aed" } modify_external_catalogs: true
I20250109 20:52:59.180560 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_33b6f13c94484781be39523332e10aed" } modify_external_catalogs: true
I20250109 20:52:59.181839 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_38191599779143059082c1ed7b3ccb76" } new_table_name: "default.Table_44be3f0582b34794be54b0e89682b0f1" modify_external_catalogs: true
I20250109 20:52:59.184845 1166 ts_tablet_manager.cc:1918] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:52:59.185683 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250109 20:52:59.185956 1158 tablet_bootstrap.cc:492] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.186065 1158 ts_tablet_manager.cc:1397] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250109 20:52:59.187018 1166 log.cc:1198] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/686ed5f7d5524c54a8debe1bc610bd25
I20250109 20:52:59.187288 1166 ts_tablet_manager.cc:1939] T 686ed5f7d5524c54a8debe1bc610bd25 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.188047 1166 tablet_replica.cc:331] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.188205 1166 raft_consensus.cc:2238] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.184937 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 801838c21f70424ba1b2d89585bda643 (DEFAULT_TABLE table=default.Table_261f03425c0e4e77873973fa3746a8c3 [id=ae99a336de5a4336bfaa8e4d2f8994e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.188594 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801838c21f70424ba1b2d89585bda643. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:59.189173 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 -> Peer 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583): Couldn't send request to peer 36a29ab2002d4eb389cd11778988e1e9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250109 20:52:59.189276 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 686ed5f7d5524c54a8debe1bc610bd25 (table default.Table_6cca94c4ba974861b95038c43ebf833a [id=bf679a467a78433899ead891f63fed13]) successfully deleted
I20250109 20:52:59.189754 892 tablet_service.cc:1467] Processing CreateTablet for tablet 578aee2c3c26404f8ef967241337d74d (DEFAULT_TABLE table=default.Table_48b3a08a9f7c4555ac5bed1df8590ebb [id=65b4c85da952464dbc4d0ce58e1387d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.190205 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578aee2c3c26404f8ef967241337d74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.190580 1157 tablet_replica.cc:331] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.190659 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 59e1e8754625463ea42a92ef65a89214 (table default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]) successfully deleted
I20250109 20:52:59.190742 1157 raft_consensus.cc:2238] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.190850 1157 raft_consensus.cc:2267] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.191296 1157 ts_tablet_manager.cc:1905] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.186738 1158 raft_consensus.cc:357] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.192869 1158 raft_consensus.cc:383] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.192934 1158 raft_consensus.cc:738] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.193164 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3a97d64313b6410788dafa912c301267" } new_table_name: "default.Table_c720f45ee7754dc3ac46aaa4cbf96871" modify_external_catalogs: true
I20250109 20:52:59.193145 1158 consensus_queue.cc:260] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.193751 1158 ts_tablet_manager.cc:1428] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250109 20:52:59.188310 1166 raft_consensus.cc:2267] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.197242 889 tablet_service.cc:1467] Processing CreateTablet for tablet 801838c21f70424ba1b2d89585bda643 (DEFAULT_TABLE table=default.Table_261f03425c0e4e77873973fa3746a8c3 [id=ae99a336de5a4336bfaa8e4d2f8994e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.197207 1158 tablet_bootstrap.cc:492] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.197652 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801838c21f70424ba1b2d89585bda643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.199942 1158 tablet_bootstrap.cc:654] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.204643 1157 ts_tablet_manager.cc:1918] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.204789 1157 log.cc:1198] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/91dbccae0af24a629600a92e7eadfee7
I20250109 20:52:59.204797 1158 tablet_bootstrap.cc:492] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.204934 1158 ts_tablet_manager.cc:1397] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250109 20:52:59.205108 1157 ts_tablet_manager.cc:1939] T 91dbccae0af24a629600a92e7eadfee7 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.205897 1158 raft_consensus.cc:357] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.206144 1158 raft_consensus.cc:383] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.206213 1158 raft_consensus.cc:738] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:59.206485 1165 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.102s sys 0.132s
W20250109 20:52:59.206583 1165 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.538s user 0.102s sys 0.132s
I20250109 20:52:59.207267 1165 tablet_bootstrap.cc:492] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:59.207443 1165 ts_tablet_manager.cc:1397] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.541s user 0.104s sys 0.133s
I20250109 20:52:59.207538 1176 tablet_bootstrap.cc:492] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.207549 1158 consensus_queue.cc:260] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.207667 1176 ts_tablet_manager.cc:1397] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.007s
I20250109 20:52:59.208313 1165 raft_consensus.cc:357] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.208415 1176 raft_consensus.cc:357] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.208535 1165 raft_consensus.cc:383] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.208612 1176 raft_consensus.cc:383] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.208709 1176 raft_consensus.cc:738] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.208762 1165 raft_consensus.cc:738] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.208978 1176 consensus_queue.cc:260] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.209100 1158 ts_tablet_manager.cc:1428] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:52:59.209074 1165 consensus_queue.cc:260] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.209843 1176 ts_tablet_manager.cc:1428] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:59.210101 1176 tablet_bootstrap.cc:492] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.210137 1166 ts_tablet_manager.cc:1905] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.211670 1176 tablet_bootstrap.cc:654] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.213841 1165 ts_tablet_manager.cc:1428] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250109 20:52:59.216485 1165 tablet_bootstrap.cc:492] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:59.216629 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 91dbccae0af24a629600a92e7eadfee7 (table default.Table_534eb61fef3c42c8bac87ac39a09ef45 [id=0e350177d27e4c8b8bc3cede2166517e]) successfully deleted
I20250109 20:52:59.217037 1157 tablet_replica.cc:331] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.217204 1157 raft_consensus.cc:2238] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.217317 1157 raft_consensus.cc:2267] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.219586 1157 ts_tablet_manager.cc:1905] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.223948 1158 tablet_bootstrap.cc:492] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.226043 1158 tablet_bootstrap.cc:654] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.229158 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 578aee2c3c26404f8ef967241337d74d (DEFAULT_TABLE table=default.Table_48b3a08a9f7c4555ac5bed1df8590ebb [id=65b4c85da952464dbc4d0ce58e1387d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.229564 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578aee2c3c26404f8ef967241337d74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.230568 747 tablet_service.cc:1467] Processing CreateTablet for tablet 14e483cd4ae64fe1aa31cb0e4b6d6338 (DEFAULT_TABLE table=default.Table_9a8af3d5811647918198f9152cac1a8a [id=cf34a067633e40758f14fda58845d370]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.230881 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e483cd4ae64fe1aa31cb0e4b6d6338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.230877 1165 tablet_bootstrap.cc:654] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.231469 1157 ts_tablet_manager.cc:1918] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.231545 1157 log.cc:1198] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/a1ed00f7954143c9ba9bf65d8a8a68d2
I20250109 20:52:59.231770 1157 ts_tablet_manager.cc:1939] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.232573 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet a1ed00f7954143c9ba9bf65d8a8a68d2 (table default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]) successfully deleted
I20250109 20:52:59.232849 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 825220629faa4d2481e0db9fa226c306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.233263 1176 tablet_bootstrap.cc:492] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.233378 1176 ts_tablet_manager.cc:1397] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250109 20:52:59.234000 1176 raft_consensus.cc:357] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.234174 1176 raft_consensus.cc:383] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.234225 1176 raft_consensus.cc:738] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.234472 1176 consensus_queue.cc:260] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.237286 740 tablet_service.cc:1467] Processing CreateTablet for tablet 578aee2c3c26404f8ef967241337d74d (DEFAULT_TABLE table=default.Table_48b3a08a9f7c4555ac5bed1df8590ebb [id=65b4c85da952464dbc4d0ce58e1387d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.237921 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578aee2c3c26404f8ef967241337d74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.240178 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5849e3d0d9be47ba897b3e55f1492c72
I20250109 20:52:59.241688 1176 ts_tablet_manager.cc:1428] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250109 20:52:59.242306 1176 tablet_bootstrap.cc:492] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:59.242554 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c708cc308b42444e9edbe7198e78a49f consensus state has no leader
I20250109 20:52:59.243933 1176 tablet_bootstrap.cc:654] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.245589 1158 tablet_bootstrap.cc:492] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.245754 1158 ts_tablet_manager.cc:1397] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250109 20:52:59.246651 1158 raft_consensus.cc:357] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.246802 1158 raft_consensus.cc:383] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.246838 1158 raft_consensus.cc:738] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.247026 1158 consensus_queue.cc:260] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.247972 1158 ts_tablet_manager.cc:1428] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:59.244954 1157 tablet_replica.cc:331] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.248116 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.248131 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d1a9999752c4415ca70dcb514d0e0003"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.248287 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet e090f72eb1e046319b825ee5fe126332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.248409 1158 tablet_bootstrap.cc:492] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:52:59.248579 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1a9999752c4415ca70dcb514d0e0003 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.249643 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da78917f0b4931ab1705371c1fe5dd with delete_type TABLET_DATA_DELETED (replaced by tablet 3974c4196d4043be9ad6e0452ed9cb7b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.250020 1158 tablet_bootstrap.cc:654] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.250639 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8c8845f0032e48d787d1e31838cc5c17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.250967 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 14e483cd4ae64fe1aa31cb0e4b6d6338 (DEFAULT_TABLE table=default.Table_9a8af3d5811647918198f9152cac1a8a [id=cf34a067633e40758f14fda58845d370]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:59.251085 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c8845f0032e48d787d1e31838cc5c17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.251297 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e483cd4ae64fe1aa31cb0e4b6d6338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.252676 1157 raft_consensus.cc:2238] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.252843 1157 raft_consensus.cc:2267] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.253273 1157 ts_tablet_manager.cc:1905] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.255369 1176 tablet_bootstrap.cc:492] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.255491 1176 ts_tablet_manager.cc:1397] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20250109 20:52:59.256103 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4" } modify_external_catalogs: true
I20250109 20:52:59.256194 1176 raft_consensus.cc:357] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.256354 1176 raft_consensus.cc:383] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.256400 1176 raft_consensus.cc:738] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.256567 1176 consensus_queue.cc:260] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.256595 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4" } modify_external_catalogs: true
I20250109 20:52:59.257189 1176 ts_tablet_manager.cc:1428] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:59.257400 1176 tablet_bootstrap.cc:492] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.259162 1176 tablet_bootstrap.cc:654] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.261503 890 tablet_service.cc:1467] Processing CreateTablet for tablet fece357e3b9543aa8ca1fc85cc3caaff (DEFAULT_TABLE table=default.Table_5e1520f094e94246921394f3b8b307df [id=bc99620b468a47ef8201b31c695ee4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.261826 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fece357e3b9543aa8ca1fc85cc3caaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.264964 1166 ts_tablet_manager.cc:1918] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.265067 1166 log.cc:1198] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/246fd9ac96f54d008e8203e4580e7ef3
I20250109 20:52:59.265581 892 tablet_service.cc:1467] Processing CreateTablet for tablet 14e483cd4ae64fe1aa31cb0e4b6d6338 (DEFAULT_TABLE table=default.Table_9a8af3d5811647918198f9152cac1a8a [id=cf34a067633e40758f14fda58845d370]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.265642 1158 tablet_bootstrap.cc:492] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.265785 1158 ts_tablet_manager.cc:1397] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250109 20:52:59.266036 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e483cd4ae64fe1aa31cb0e4b6d6338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.266487 1158 raft_consensus.cc:357] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.266644 1158 raft_consensus.cc:383] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.266683 1158 raft_consensus.cc:738] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.266865 1158 consensus_queue.cc:260] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.267969 1158 ts_tablet_manager.cc:1428] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:52:59.268828 1158 tablet_bootstrap.cc:492] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.268831 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:59.269057 1160 tablet_replica.cc:331] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:59.269239 1160 raft_consensus.cc:2238] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.269410 1160 raft_consensus.cc:2267] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.270306 1160 ts_tablet_manager.cc:1905] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.273649 755 tablet_service.cc:1467] Processing CreateTablet for tablet fece357e3b9543aa8ca1fc85cc3caaff (DEFAULT_TABLE table=default.Table_5e1520f094e94246921394f3b8b307df [id=bc99620b468a47ef8201b31c695ee4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.273977 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fece357e3b9543aa8ca1fc85cc3caaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.275696 1160 ts_tablet_manager.cc:1918] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.275781 1160 log.cc:1198] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4015c0e66dfd47bfb9855f82044f01e7
I20250109 20:52:59.276113 1160 ts_tablet_manager.cc:1939] T 4015c0e66dfd47bfb9855f82044f01e7 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:59.276983 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4015c0e66dfd47bfb9855f82044f01e7 (table default.Table_532d28a5835d421ab3f13d74939ec028 [id=2a15fab6f69a4e399c87563e7e5be289]) successfully deleted
I20250109 20:52:59.278277 1158 tablet_bootstrap.cc:654] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.279846 1157 ts_tablet_manager.cc:1918] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.279951 1157 log.cc:1198] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/8a5a6dbb823e4bd3afeb157b3c5771ba
I20250109 20:52:59.280225 1157 ts_tablet_manager.cc:1939] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.280694 1165 tablet_bootstrap.cc:492] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:59.281868 2129 raft_consensus.cc:491] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.281931 1165 ts_tablet_manager.cc:1397] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.002s
I20250109 20:52:59.282004 2129 raft_consensus.cc:513] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.282656 2129 leader_election.cc:290] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.283015 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e090f72eb1e046319b825ee5fe126332" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.283171 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e090f72eb1e046319b825ee5fe126332" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.283581 990 leader_election.cc:343] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.283754 989 leader_election.cc:343] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.283845 989 leader_election.cc:304] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.284204 2129 raft_consensus.cc:2743] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.284119 1165 raft_consensus.cc:357] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.286223 1165 raft_consensus.cc:383] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.286346 1165 raft_consensus.cc:738] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.286058 889 tablet_service.cc:1467] Processing CreateTablet for tablet 81db8c605d8944c89d9184b81de8e5d6 (DEFAULT_TABLE table=default.Table_ae7ad8e7e47a4b3fa7011f6db7700313 [id=116eadc3ef1c42ec85ef73873a9bf69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.286617 739 tablet_service.cc:1467] Processing CreateTablet for tablet 81db8c605d8944c89d9184b81de8e5d6 (DEFAULT_TABLE table=default.Table_ae7ad8e7e47a4b3fa7011f6db7700313 [id=116eadc3ef1c42ec85ef73873a9bf69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.286921 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81db8c605d8944c89d9184b81de8e5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.286921 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81db8c605d8944c89d9184b81de8e5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.289234 1020 tablet_service.cc:1467] Processing CreateTablet for tablet fece357e3b9543aa8ca1fc85cc3caaff (DEFAULT_TABLE table=default.Table_5e1520f094e94246921394f3b8b307df [id=bc99620b468a47ef8201b31c695ee4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.289568 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fece357e3b9543aa8ca1fc85cc3caaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.289506 1165 consensus_queue.cc:260] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.290303 1678 consensus_queue.cc:1035] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:52:59.291591 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 81db8c605d8944c89d9184b81de8e5d6 (DEFAULT_TABLE table=default.Table_ae7ad8e7e47a4b3fa7011f6db7700313 [id=116eadc3ef1c42ec85ef73873a9bf69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.291862 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81db8c605d8944c89d9184b81de8e5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.292730 890 tablet_service.cc:1467] Processing CreateTablet for tablet 46a90f1d33184c45be759f5c2bbe5a77 (DEFAULT_TABLE table=default.Table_d6c47101ffbb45b8b62f228bfa7720e7 [id=7bf38d4816384492812e03210df32651]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.293031 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a90f1d33184c45be759f5c2bbe5a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.294332 750 tablet_service.cc:1467] Processing CreateTablet for tablet 46a90f1d33184c45be759f5c2bbe5a77 (DEFAULT_TABLE table=default.Table_d6c47101ffbb45b8b62f228bfa7720e7 [id=7bf38d4816384492812e03210df32651]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.294633 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a90f1d33184c45be759f5c2bbe5a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.294871 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 46a90f1d33184c45be759f5c2bbe5a77 (DEFAULT_TABLE table=default.Table_d6c47101ffbb45b8b62f228bfa7720e7 [id=7bf38d4816384492812e03210df32651]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.295167 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a90f1d33184c45be759f5c2bbe5a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.296252 745 tablet_service.cc:1467] Processing CreateTablet for tablet 670375bfd5114040aed6be25d1374d42 (DEFAULT_TABLE table=default.Table_a997c69220da46df9489620287286652 [id=ffea19003ebc447d964b2ac3ea3a4829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.296521 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670375bfd5114040aed6be25d1374d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.296944 896 tablet_service.cc:1467] Processing CreateTablet for tablet d3428128a5494341b1986ce6956bb88a (DEFAULT_TABLE table=default.Table_1326ea6538f74cb28d688c06eda263df [id=2aa19c381d594d3fa83c0670db671645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.297216 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3428128a5494341b1986ce6956bb88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.300141 1026 tablet_service.cc:1467] Processing CreateTablet for tablet d3428128a5494341b1986ce6956bb88a (DEFAULT_TABLE table=default.Table_1326ea6538f74cb28d688c06eda263df [id=2aa19c381d594d3fa83c0670db671645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.300452 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3428128a5494341b1986ce6956bb88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.300943 756 tablet_service.cc:1467] Processing CreateTablet for tablet d3428128a5494341b1986ce6956bb88a (DEFAULT_TABLE table=default.Table_1326ea6538f74cb28d688c06eda263df [id=2aa19c381d594d3fa83c0670db671645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.301244 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3428128a5494341b1986ce6956bb88a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.301417 758 tablet_service.cc:1467] Processing CreateTablet for tablet 086c6b621cae4214bcec4daa1a725272 (DEFAULT_TABLE table=default.Table_613b9c78c2024fb58046e80d8ad29df8 [id=14668d1928ee441da233185d8d49d9a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.301690 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086c6b621cae4214bcec4daa1a725272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.302349 899 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8e0351fe654fa68e31e4308beb1c79 (DEFAULT_TABLE table=default.Table_e46707392eee49959bca8c498cf57a76 [id=6e55c5cfc56944298609747d067e98fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.302747 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a8e0351fe654fa68e31e4308beb1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.304172 897 tablet_service.cc:1467] Processing CreateTablet for tablet 086c6b621cae4214bcec4daa1a725272 (DEFAULT_TABLE table=default.Table_613b9c78c2024fb58046e80d8ad29df8 [id=14668d1928ee441da233185d8d49d9a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.304427 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086c6b621cae4214bcec4daa1a725272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.304409 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 086c6b621cae4214bcec4daa1a725272 (DEFAULT_TABLE table=default.Table_613b9c78c2024fb58046e80d8ad29df8 [id=14668d1928ee441da233185d8d49d9a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.304673 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086c6b621cae4214bcec4daa1a725272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.304877 2129 raft_consensus.cc:491] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.305002 891 tablet_service.cc:1467] Processing CreateTablet for tablet 670375bfd5114040aed6be25d1374d42 (DEFAULT_TABLE table=default.Table_a997c69220da46df9489620287286652 [id=ffea19003ebc447d964b2ac3ea3a4829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.305002 2129 raft_consensus.cc:513] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.305296 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670375bfd5114040aed6be25d1374d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.305537 2129 leader_election.cc:290] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.305899 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825220629faa4d2481e0db9fa226c306" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.306042 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825220629faa4d2481e0db9fa226c306" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.306427 989 leader_election.cc:343] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.306551 990 leader_election.cc:343] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.306625 990 leader_election.cc:304] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.306802 2129 raft_consensus.cc:2743] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:52:59.315203 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet be05de223c3946c6b8a4899cf696fe26 (table default.Table_17aab5c8aa9a4999b2eebd66dcae9f46 [id=3bdc7c0be0d14815a92667bde1011e67], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet be05de223c3946c6b8a4899cf696fe26 consensus state has no leader
I20250109 20:52:59.300463 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 670375bfd5114040aed6be25d1374d42 (DEFAULT_TABLE table=default.Table_a997c69220da46df9489620287286652 [id=ffea19003ebc447d964b2ac3ea3a4829]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.316488 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670375bfd5114040aed6be25d1374d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.317473 1678 raft_consensus.cc:491] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.317624 1678 raft_consensus.cc:513] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.318270 1678 leader_election.cc:290] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:59.318902 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d2ad301c11943b69b59be18d6775513" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.319429 712 leader_election.cc:343] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.320271 713 leader_election.cc:343] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 4d2ad301c11943b69b59be18d6775513
I20250109 20:52:59.320379 713 leader_election.cc:304] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:59.320825 1678 raft_consensus.cc:2743] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.321756 1165 ts_tablet_manager.cc:1428] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.039s user 0.001s sys 0.000s
I20250109 20:52:59.325389 1165 tablet_bootstrap.cc:492] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:59.329849 1165 tablet_bootstrap.cc:654] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.342113 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9c88f42c15d94099903178218bfac1e5
I20250109 20:52:59.343643 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c88f42c15d94099903178218bfac1e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.344182 2219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:41583 every 8 connection(s)
I20250109 20:52:59.344589 1922 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:41583
W20250109 20:52:59.344974 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6fa1245ccea427eb6b2c0eaadebd24e consensus state has no leader
I20250109 20:52:59.345347 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_eb7bd4795ee740f0ba9088dc52e7b383" } modify_external_catalogs: true
I20250109 20:52:59.345527 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_eb7bd4795ee740f0ba9088dc52e7b383" } modify_external_catalogs: true
I20250109 20:52:59.346776 1166 ts_tablet_manager.cc:1939] T 246fd9ac96f54d008e8203e4580e7ef3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.347054 1922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
I20250109 20:52:59.347097 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5157484d0a55469184fc46834bcf66bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.347733 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1c5cef087cc349eba40c171b326a1f8f
I20250109 20:52:59.348367 1176 tablet_bootstrap.cc:492] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.348573 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c88f42c15d94099903178218bfac1e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.348762 1176 ts_tablet_manager.cc:1397] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.008s
I20250109 20:52:59.350214 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu as pid 1922
I20250109 20:52:59.353816 1157 tablet_replica.cc:331] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.354075 1157 raft_consensus.cc:2238] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.354219 1157 raft_consensus.cc:2267] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.354820 1157 ts_tablet_manager.cc:1905] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.356904 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c88f42c15d94099903178218bfac1e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:59.357599 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9c88f42c15d94099903178218bfac1e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c88f42c15d94099903178218bfac1e5 already in progress: creating tablet
I20250109 20:52:59.357884 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 4015c0e66dfd47bfb9855f82044f01e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:52:59.359227 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4015c0e66dfd47bfb9855f82044f01e7 because the tablet was not found. No further retry: Not found: Tablet not found: 4015c0e66dfd47bfb9855f82044f01e7
I20250109 20:52:59.362666 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 0019a01afbcc452a9a9374c92dc131f3 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:59.348003 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 246fd9ac96f54d008e8203e4580e7ef3 (table default.Table_8ed167dde7b94790b7731c10aa30628c [id=87a978ea0af846f0b3ea56a3d1942d6d]) successfully deleted
I20250109 20:52:59.364037 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 8a5a6dbb823e4bd3afeb157b3c5771ba (table default.Table_e677fcf0892746659b8f6d9f5dce6bc1 [id=d8d809ef804f4f909b825f92a823fb8d]) successfully deleted
I20250109 20:52:59.364857 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f85944f66543fe8dd634ad0da0728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:59.365585 712 consensus_peers.cc:487] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250109 20:52:59.365866 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5157484d0a55469184fc46834bcf66bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.366880 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f420b7e623243e093d26c95276c5af9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2420b285905342b3820f9549c8f0c4ad) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.367046 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5cef087cc349eba40c171b326a1f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.367780 745 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.367904 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.368012 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5cef087cc349eba40c171b326a1f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.368033 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 2b1891cbef8f4b0cae0c15d478db4860 (table default.Table_ca51464e265c4e28b9e54fe7be0587fc [id=0a0599668d77405ba0e72eda2aa5219e]). Expected version 4 got 0
I20250109 20:52:59.371479 896 tablet_service.cc:1514] Processing DeleteTablet for tablet b221e048f6f6463e84e630386c304a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:59.371735 1160 tablet_replica.cc:331] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:59.371945 1160 raft_consensus.cc:2238] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.372113 1160 raft_consensus.cc:2267] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.373042 1160 ts_tablet_manager.cc:1905] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.349707 1176 raft_consensus.cc:357] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.373191 1176 raft_consensus.cc:383] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.373281 1176 raft_consensus.cc:738] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.373966 1176 consensus_queue.cc:260] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.376807 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5cef087cc349eba40c171b326a1f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:59.377058 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8e0351fe654fa68e31e4308beb1c79 (DEFAULT_TABLE table=default.Table_e46707392eee49959bca8c498cf57a76 [id=6e55c5cfc56944298609747d067e98fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.377405 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a8e0351fe654fa68e31e4308beb1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.377833 890 tablet_service.cc:1467] Processing CreateTablet for tablet ceb2a882918046af95a884b87b0d084f (DEFAULT_TABLE table=default.Table_baf2e0ecb9594bc3af2f86ba420846d4 [id=8ebdd3108bfd4995905f2e90c646d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.378157 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb2a882918046af95a884b87b0d084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.379447 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d88561e6adcd4340a382661b36372ebc (DEFAULT_TABLE table=default.Table_894a7c1fbfb54931992683e6feb51276 [id=471987a71eb240c98b406089e93ef36b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.379667 1176 ts_tablet_manager.cc:1428] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250109 20:52:59.379734 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88561e6adcd4340a382661b36372ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.380913 896 tablet_service.cc:1467] Processing CreateTablet for tablet d88561e6adcd4340a382661b36372ebc (DEFAULT_TABLE table=default.Table_894a7c1fbfb54931992683e6feb51276 [id=471987a71eb240c98b406089e93ef36b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.381210 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88561e6adcd4340a382661b36372ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.382728 1016 tablet_service.cc:1467] Processing CreateTablet for tablet ceb2a882918046af95a884b87b0d084f (DEFAULT_TABLE table=default.Table_baf2e0ecb9594bc3af2f86ba420846d4 [id=8ebdd3108bfd4995905f2e90c646d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.383016 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb2a882918046af95a884b87b0d084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.384343 1176 tablet_bootstrap.cc:492] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.385463 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8d67b624984a0bb327d045c55476da with delete_type TABLET_DATA_DELETED (replaced by tablet 687db3e734af47339c2546970a71a539) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.385697 746 tablet_service.cc:1467] Processing CreateTablet for tablet d88561e6adcd4340a382661b36372ebc (DEFAULT_TABLE table=default.Table_894a7c1fbfb54931992683e6feb51276 [id=471987a71eb240c98b406089e93ef36b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.385804 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca83a49523b45be9ac7556b40746a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.388459 758 tablet_service.cc:1467] Processing CreateTablet for tablet ceb2a882918046af95a884b87b0d084f (DEFAULT_TABLE table=default.Table_baf2e0ecb9594bc3af2f86ba420846d4 [id=8ebdd3108bfd4995905f2e90c646d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.389997 1176 tablet_bootstrap.cc:654] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.390460 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.390928 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d88561e6adcd4340a382661b36372ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.390892 745 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7d7695ff5946abb6ee576431d2e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.391119 755 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8e0351fe654fa68e31e4308beb1c79 (DEFAULT_TABLE table=default.Table_e46707392eee49959bca8c498cf57a76 [id=6e55c5cfc56944298609747d067e98fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.391261 1166 tablet_replica.cc:331] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.391430 1166 raft_consensus.cc:2238] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.391469 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a8e0351fe654fa68e31e4308beb1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.391546 1166 raft_consensus.cc:2267] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.391968 1166 ts_tablet_manager.cc:1905] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.393949 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_75f8ca17cad34737a386cc58f15a0697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.394433 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75f8ca17cad34737a386cc58f15a0697 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.395026 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb2a882918046af95a884b87b0d084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.403761 1160 ts_tablet_manager.cc:1918] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.403929 1160 log.cc:1198] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/b221e048f6f6463e84e630386c304a05
I20250109 20:52:59.404289 1160 ts_tablet_manager.cc:1939] T b221e048f6f6463e84e630386c304a05 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:52:59.405543 1176 tablet_bootstrap.cc:492] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.405671 1176 ts_tablet_manager.cc:1397] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250109 20:52:59.406438 1176 raft_consensus.cc:357] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.406615 1176 raft_consensus.cc:383] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.406659 1176 raft_consensus.cc:738] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.406838 1176 consensus_queue.cc:260] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.408767 1157 ts_tablet_manager.cc:1918] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.408936 1157 log.cc:1198] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/47726b58d6714cd1b46848a16a34ff37
I20250109 20:52:59.409318 1157 ts_tablet_manager.cc:1939] T 47726b58d6714cd1b46848a16a34ff37 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.410362 1157 tablet_replica.cc:331] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.410595 1157 raft_consensus.cc:2238] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.410759 1157 raft_consensus.cc:2267] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.411366 1157 ts_tablet_manager.cc:1905] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.415380 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 47726b58d6714cd1b46848a16a34ff37 (table default.Table_f33bf5f9431146a5b111b0448a62e85d [id=3123de0f3f7644739ee589a4c7e9edec]) successfully deleted
I20250109 20:52:59.420188 1157 ts_tablet_manager.cc:1918] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.420308 1157 log.cc:1198] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4d2ad301c11943b69b59be18d6775513
I20250109 20:52:59.420629 1157 ts_tablet_manager.cc:1939] T 4d2ad301c11943b69b59be18d6775513 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.421180 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:52:59.423102 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c785c74ecee84e6fb5d27560cc7c6f28 consensus state has no leader
I20250109 20:52:59.423319 2223 sys_catalog.cc:263] Verifying existing consensus state
I20250109 20:52:59.424959 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7336cda1ce164722949bdefb9f26e7cd" } new_table_name: "default.Table_915d920262ad443daadf80f5f7367d69" modify_external_catalogs: true
W20250109 20:52:59.426146 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 52b555c0449e4ed280c9116b151b3b7a has no consensus state
I20250109 20:52:59.426417 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d1574f6ea52544dd8cc0cec65fd89000"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.426837 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1574f6ea52544dd8cc0cec65fd89000 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.427976 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_80c4e31485334c5787b69339099d015c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.428386 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80c4e31485334c5787b69339099d015c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.429620 1158 tablet_bootstrap.cc:492] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.429613 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d9f9c4769d054758bcfc87654ec86bfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.429759 1158 ts_tablet_manager.cc:1397] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.162s user 0.005s sys 0.024s
W20250109 20:52:59.429991 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9f9c4769d054758bcfc87654ec86bfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.431010 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f672cbd6dffc49dbb69992ae7db979ea
I20250109 20:52:59.430955 755 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb88771d2b148bdae6e427c1b25b9a9 (DEFAULT_TABLE table=default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.431298 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cb88771d2b148bdae6e427c1b25b9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.431525 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7f04a87f321a4355bb2fc4cbfb0b9076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.431878 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f04a87f321a4355bb2fc4cbfb0b9076 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.434288 2129 raft_consensus.cc:491] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.434405 2129 raft_consensus.cc:513] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.434976 2129 leader_election.cc:290] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.435555 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb88771d2b148bdae6e427c1b25b9a9 (DEFAULT_TABLE table=default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.435675 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845bac2539e478083bf7fe6cbc0c717" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.435904 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cb88771d2b148bdae6e427c1b25b9a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:59.436179 990 leader_election.cc:343] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.437144 891 tablet_service.cc:1467] Processing CreateTablet for tablet b67810d55d744031a81c177ec07e3cfe (DEFAULT_TABLE table=default.Table_af24f0c294ec48e7ba9e760a802a7696 [id=2fb050da26d248e4a40ce27d5e50667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.437490 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67810d55d744031a81c177ec07e3cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.437573 885 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0059863d73460a9e7d12c46a7616f3 (DEFAULT_TABLE table=default.Table_29a0e6a6805a4ffa952d1bd55e846a54 [id=9ae45a6eacd94c7e9c28a3c8f499c5fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.437880 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0059863d73460a9e7d12c46a7616f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.438809 892 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb88771d2b148bdae6e427c1b25b9a9 (DEFAULT_TABLE table=default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.439092 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cb88771d2b148bdae6e427c1b25b9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.440938 880 tablet_service.cc:1467] Processing CreateTablet for tablet 795f0f29f0744f4e9dd2fcf1762ab978 (DEFAULT_TABLE table=default.Table_385822fe7219471bacdd1ad2dbe7d891 [id=0430588a51ed49ae86290eaab54f79dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.441220 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795f0f29f0744f4e9dd2fcf1762ab978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.443994 899 tablet_service.cc:1514] Processing DeleteTablet for tablet f672cbd6dffc49dbb69992ae7db979ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:59.444949 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb65a1eb4cc435385add73ad0c3ca9d (DEFAULT_TABLE table=default.Table_3beb193926bd4164a21137f32bf717ac [id=c26de09aefb246f3a7a8b34c1e38f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.445243 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb65a1eb4cc435385add73ad0c3ca9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.448351 1158 raft_consensus.cc:357] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.448539 1158 raft_consensus.cc:383] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.448598 1158 raft_consensus.cc:738] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.448943 887 tablet_service.cc:1467] Processing CreateTablet for tablet 530369cc998a4e9980eed6d4b1f57ad4 (DEFAULT_TABLE table=default.Table_07a5579463ed4880b5f06a89dfb2c469 [id=8a0b7f55f1f34ca28c5714034fd0aeba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.448913 1158 consensus_queue.cc:260] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.449249 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530369cc998a4e9980eed6d4b1f57ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.449757 2223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Bootstrap starting.
I20250109 20:52:59.450994 1157 tablet_replica.cc:331] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.451166 1157 raft_consensus.cc:2238] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.451292 1157 raft_consensus.cc:2267] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.451750 1157 ts_tablet_manager.cc:1905] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.452951 894 tablet_service.cc:1467] Processing CreateTablet for tablet 8f933f9582c24c63b9893da59e5a66d0 (DEFAULT_TABLE table=default.Table_19300a35d595402fbb4d39933c3ba00c [id=3deabb07f6e44f06a2d58f30eb4009bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.453231 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f933f9582c24c63b9893da59e5a66d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.453572 1158 ts_tablet_manager.cc:1428] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.024s user 0.000s sys 0.003s
I20250109 20:52:59.454005 1176 ts_tablet_manager.cc:1428] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.048s user 0.002s sys 0.001s
I20250109 20:52:59.454301 1176 tablet_bootstrap.cc:492] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.455838 1176 tablet_bootstrap.cc:654] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.456935 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5923c19b7c8443eb85b66baa2bb7ffbb (DEFAULT_TABLE table=default.Table_cf82774c3759452ba2e9bde7ddc84311 [id=a00598da070c469980537525e799ea17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.457208 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5923c19b7c8443eb85b66baa2bb7ffbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.458948 1166 ts_tablet_manager.cc:1918] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.459026 1166 log.cc:1198] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/825220629faa4d2481e0db9fa226c306
I20250109 20:52:59.459254 1166 ts_tablet_manager.cc:1939] T 825220629faa4d2481e0db9fa226c306 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.459874 1158 tablet_bootstrap.cc:492] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.460955 883 tablet_service.cc:1467] Processing CreateTablet for tablet e427cb835c8140bd8d28050b9f1c4399 (DEFAULT_TABLE table=default.Table_31f8ab4bd8064cbb97ea27b925f31e33 [id=8a804fd425684a938e3196b9d737ffaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.461222 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e427cb835c8140bd8d28050b9f1c4399. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.461381 1158 tablet_bootstrap.cc:654] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.465977 2129 raft_consensus.cc:491] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.466099 2129 raft_consensus.cc:513] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.466157 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845bac2539e478083bf7fe6cbc0c717" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.466665 2129 leader_election.cc:290] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.467348 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80da78917f0b4931ab1705371c1fe5dd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:59.468348 990 leader_election.cc:343] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.469640 989 leader_election.cc:343] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.469735 989 leader_election.cc:304] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.470111 2129 raft_consensus.cc:2743] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.470322 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80da78917f0b4931ab1705371c1fe5dd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.472822 989 leader_election.cc:343] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.472900 989 leader_election.cc:304] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.473242 2129 raft_consensus.cc:2743] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.473634 897 tablet_service.cc:1467] Processing CreateTablet for tablet 38ebd03a33b94fb0946a56d25c7ab1a6 (DEFAULT_TABLE table=default.Table_4b5c0e0e5bfa4599ab2035fa66341d45 [id=863d92969d9d4800a7bfe47431ea2123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.473951 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ebd03a33b94fb0946a56d25c7ab1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.475507 1176 tablet_bootstrap.cc:492] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.475616 1176 ts_tablet_manager.cc:1397] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20250109 20:52:59.476300 1176 raft_consensus.cc:357] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.476442 1176 raft_consensus.cc:383] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.476485 1176 raft_consensus.cc:738] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.476660 1176 consensus_queue.cc:260] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.481160 1176 ts_tablet_manager.cc:1428] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:52:59.481451 1176 tablet_bootstrap.cc:492] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.482780 1176 tablet_bootstrap.cc:654] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.484280 1158 tablet_bootstrap.cc:492] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.484448 1158 ts_tablet_manager.cc:1397] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250109 20:52:59.485152 1158 raft_consensus.cc:357] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.485339 1158 raft_consensus.cc:383] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.485394 1158 raft_consensus.cc:738] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.485585 1158 consensus_queue.cc:260] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.487768 1158 ts_tablet_manager.cc:1428] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:59.488018 1158 tablet_bootstrap.cc:492] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.489583 1158 tablet_bootstrap.cc:654] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.494752 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:59.496531 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1779840e861f4e7caab371208ae15c40 (DEFAULT_TABLE table=default.Table_213216398c104dca9ffe9a505d0e46fe [id=dd6db2a66af5463698edf91dd88d700e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.496614 1166 tablet_replica.cc:331] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.496829 1166 raft_consensus.cc:2238] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.497007 1166 raft_consensus.cc:2267] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.497622 755 tablet_service.cc:1514] Processing DeleteTablet for tablet f672cbd6dffc49dbb69992ae7db979ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.498531 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet f672cbd6dffc49dbb69992ae7db979ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.499287 755 tablet_service.cc:1467] Processing CreateTablet for tablet b087b7546a39451089666a36d029e645 (DEFAULT_TABLE table=default.Table_1b296a6b3a0c47eb8dead9e581ac987b [id=9bda551fac0244c7bd079d717596657d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.499488 1166 ts_tablet_manager.cc:1905] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.499598 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b087b7546a39451089666a36d029e645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.503998 2129 raft_consensus.cc:491] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.504130 2129 raft_consensus.cc:513] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.504813 2129 leader_election.cc:290] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.507611 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1779840e861f4e7caab371208ae15c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.507593 1016 tablet_service.cc:1467] Processing CreateTablet for tablet b087b7546a39451089666a36d029e645 (DEFAULT_TABLE table=default.Table_1b296a6b3a0c47eb8dead9e581ac987b [id=9bda551fac0244c7bd079d717596657d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.497009 885 tablet_service.cc:1467] Processing CreateTablet for tablet d6fa1245ccea427eb6b2c0eaadebd24e (DEFAULT_TABLE table=default.Table_7147fbe7700d4c52a663ea8345c0a31d [id=d69be15c9d4347bd957fee13dc93724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.507903 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b087b7546a39451089666a36d029e645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.507967 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fa1245ccea427eb6b2c0eaadebd24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.498673 880 tablet_service.cc:1467] Processing CreateTablet for tablet 5b17962015e04c6b85169915586334cc (DEFAULT_TABLE table=default.Table_c0d1dadec3324eee8cc42a24ab27f3ed [id=891c9e371ac14ed6ab56b4419c45f4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.506481 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.508141 883 tablet_service.cc:1467] Processing CreateTablet for tablet 733c7059c750463aa586d9b733f1d1e9 (DEFAULT_TABLE table=default.Table_24614ad1648f41d9bc4bb77eb2b170ae [id=d452950a44b94715a2fdee7a65cba1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.508391 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733c7059c750463aa586d9b733f1d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:59.508591 989 leader_election.cc:343] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.508834 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b17962015e04c6b85169915586334cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.513520 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 38ebd03a33b94fb0946a56d25c7ab1a6 (DEFAULT_TABLE table=default.Table_4b5c0e0e5bfa4599ab2035fa66341d45 [id=863d92969d9d4800a7bfe47431ea2123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.513898 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ebd03a33b94fb0946a56d25c7ab1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.513878 1020 tablet_service.cc:1467] Processing CreateTablet for tablet b67810d55d744031a81c177ec07e3cfe (DEFAULT_TABLE table=default.Table_af24f0c294ec48e7ba9e760a802a7696 [id=2fb050da26d248e4a40ce27d5e50667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.514176 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67810d55d744031a81c177ec07e3cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.521456 1678 raft_consensus.cc:491] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.521634 882 tablet_service.cc:1467] Processing CreateTablet for tablet 94ed3ab88fd8470bbbc27d801323dc9d (DEFAULT_TABLE table=default.Table_30b1ffb8aacc48c6883ed04b539160e7 [id=eb5ba91d8d3a4f959e2b85bb77a154cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.521944 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ed3ab88fd8470bbbc27d801323dc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.522320 894 tablet_service.cc:1467] Processing CreateTablet for tablet dcba62cbadf04761822bf7161c52b758 (DEFAULT_TABLE table=default.Table_ea166542339a4980aea4afc8f128edd5 [id=2172881080574ac683ba26588c37cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.522586 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcba62cbadf04761822bf7161c52b758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.523216 887 tablet_service.cc:1467] Processing CreateTablet for tablet 88cd7f9808d547ca80a76e4b9568cd0a (DEFAULT_TABLE table=default.Table_9db2524d4be648e4828df22a952c1cb4 [id=f60e6d2c78664f3da51dc6cf6a1a9d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.523525 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88cd7f9808d547ca80a76e4b9568cd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.524509 1166 ts_tablet_manager.cc:1918] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.528613 1166 log.cc:1198] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b68d23fb2df743f4afa2682f6d2e3efa
I20250109 20:52:59.525282 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0059863d73460a9e7d12c46a7616f3 (DEFAULT_TABLE table=default.Table_29a0e6a6805a4ffa952d1bd55e846a54 [id=9ae45a6eacd94c7e9c28a3c8f499c5fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.529299 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0059863d73460a9e7d12c46a7616f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.529225 746 tablet_service.cc:1467] Processing CreateTablet for tablet b67810d55d744031a81c177ec07e3cfe (DEFAULT_TABLE table=default.Table_af24f0c294ec48e7ba9e760a802a7696 [id=2fb050da26d248e4a40ce27d5e50667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.529248 1678 raft_consensus.cc:513] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.529722 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67810d55d744031a81c177ec07e3cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.529749 758 tablet_service.cc:1467] Processing CreateTablet for tablet 38ebd03a33b94fb0946a56d25c7ab1a6 (DEFAULT_TABLE table=default.Table_4b5c0e0e5bfa4599ab2035fa66341d45 [id=863d92969d9d4800a7bfe47431ea2123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.530086 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.530464 756 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0059863d73460a9e7d12c46a7616f3 (DEFAULT_TABLE table=default.Table_29a0e6a6805a4ffa952d1bd55e846a54 [id=9ae45a6eacd94c7e9c28a3c8f499c5fc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:52:59.530555 990 leader_election.cc:343] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.530651 990 leader_election.cc:304] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.531010 2129 raft_consensus.cc:2743] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.531157 1157 ts_tablet_manager.cc:1918] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.531391 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ebd03a33b94fb0946a56d25c7ab1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.531407 1678 leader_election.cc:290] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:59.531678 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8d67b624984a0bb327d045c55476da" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.532094 712 leader_election.cc:343] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.532824 713 leader_election.cc:343] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 3d8d67b624984a0bb327d045c55476da
I20250109 20:52:59.532934 713 leader_election.cc:304] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:52:59.533232 755 tablet_service.cc:1467] Processing CreateTablet for tablet 795f0f29f0744f4e9dd2fcf1762ab978 (DEFAULT_TABLE table=default.Table_385822fe7219471bacdd1ad2dbe7d891 [id=0430588a51ed49ae86290eaab54f79dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.533510 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795f0f29f0744f4e9dd2fcf1762ab978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.534399 1984 raft_consensus.cc:2743] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.536530 1157 log.cc:1198] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/06c3aa20539740cba4e5d6e852ad2909
I20250109 20:52:59.536882 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0059863d73460a9e7d12c46a7616f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.537003 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 795f0f29f0744f4e9dd2fcf1762ab978 (DEFAULT_TABLE table=default.Table_385822fe7219471bacdd1ad2dbe7d891 [id=0430588a51ed49ae86290eaab54f79dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.537366 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795f0f29f0744f4e9dd2fcf1762ab978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.541234 758 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb65a1eb4cc435385add73ad0c3ca9d (DEFAULT_TABLE table=default.Table_3beb193926bd4164a21137f32bf717ac [id=c26de09aefb246f3a7a8b34c1e38f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.541548 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb65a1eb4cc435385add73ad0c3ca9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.542179 746 tablet_service.cc:1467] Processing CreateTablet for tablet 530369cc998a4e9980eed6d4b1f57ad4 (DEFAULT_TABLE table=default.Table_07a5579463ed4880b5f06a89dfb2c469 [id=8a0b7f55f1f34ca28c5714034fd0aeba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.542491 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530369cc998a4e9980eed6d4b1f57ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.542948 745 tablet_service.cc:1467] Processing CreateTablet for tablet 8f933f9582c24c63b9893da59e5a66d0 (DEFAULT_TABLE table=default.Table_19300a35d595402fbb4d39933c3ba00c [id=3deabb07f6e44f06a2d58f30eb4009bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.543228 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f933f9582c24c63b9893da59e5a66d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.545517 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 530369cc998a4e9980eed6d4b1f57ad4 (DEFAULT_TABLE table=default.Table_07a5579463ed4880b5f06a89dfb2c469 [id=8a0b7f55f1f34ca28c5714034fd0aeba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.545881 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 530369cc998a4e9980eed6d4b1f57ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.546393 739 tablet_service.cc:1467] Processing CreateTablet for tablet 5923c19b7c8443eb85b66baa2bb7ffbb (DEFAULT_TABLE table=default.Table_cf82774c3759452ba2e9bde7ddc84311 [id=a00598da070c469980537525e799ea17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.529023 1166 ts_tablet_manager.cc:1939] T b68d23fb2df743f4afa2682f6d2e3efa P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.546638 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5923c19b7c8443eb85b66baa2bb7ffbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.547900 2129 raft_consensus.cc:491] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.547992 2129 raft_consensus.cc:513] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.548540 2129 leader_election.cc:290] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.548946 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5923c19b7c8443eb85b66baa2bb7ffbb (DEFAULT_TABLE table=default.Table_cf82774c3759452ba2e9bde7ddc84311 [id=a00598da070c469980537525e799ea17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.549216 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5923c19b7c8443eb85b66baa2bb7ffbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.549422 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8419d8cc7e475ea43dd263cae87aca" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.549885 989 leader_election.cc:343] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.552817 753 tablet_service.cc:1467] Processing CreateTablet for tablet e427cb835c8140bd8d28050b9f1c4399 (DEFAULT_TABLE table=default.Table_31f8ab4bd8064cbb97ea27b925f31e33 [id=8a804fd425684a938e3196b9d737ffaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.553049 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 8f933f9582c24c63b9893da59e5a66d0 (DEFAULT_TABLE table=default.Table_19300a35d595402fbb4d39933c3ba00c [id=3deabb07f6e44f06a2d58f30eb4009bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.553195 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e427cb835c8140bd8d28050b9f1c4399. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.553304 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f933f9582c24c63b9893da59e5a66d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.553838 1157 ts_tablet_manager.cc:1939] T 06c3aa20539740cba4e5d6e852ad2909 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.556972 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8419d8cc7e475ea43dd263cae87aca" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.558305 1157 tablet_replica.cc:331] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:52:59.558363 990 leader_election.cc:343] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.558475 990 leader_election.cc:304] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.558688 1157 raft_consensus.cc:2238] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.558835 1157 raft_consensus.cc:2267] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.558863 2129 raft_consensus.cc:2743] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.560297 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e427cb835c8140bd8d28050b9f1c4399 (DEFAULT_TABLE table=default.Table_31f8ab4bd8064cbb97ea27b925f31e33 [id=8a804fd425684a938e3196b9d737ffaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.560556 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e427cb835c8140bd8d28050b9f1c4399. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.561930 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 1779840e861f4e7caab371208ae15c40 (DEFAULT_TABLE table=default.Table_213216398c104dca9ffe9a505d0e46fe [id=dd6db2a66af5463698edf91dd88d700e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.562237 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1779840e861f4e7caab371208ae15c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.566315 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb65a1eb4cc435385add73ad0c3ca9d (DEFAULT_TABLE table=default.Table_3beb193926bd4164a21137f32bf717ac [id=c26de09aefb246f3a7a8b34c1e38f8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.566637 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eb65a1eb4cc435385add73ad0c3ca9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.569397 1157 ts_tablet_manager.cc:1905] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.569856 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 825220629faa4d2481e0db9fa226c306 (table default.Table_4560626d5c8547678952bb1e7da86ad4 [id=f4c5a8b765814c37882dfdeaf55481e2]) successfully deleted
I20250109 20:52:59.571081 1166 tablet_replica.cc:331] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.571158 739 tablet_service.cc:1467] Processing CreateTablet for tablet 1779840e861f4e7caab371208ae15c40 (DEFAULT_TABLE table=default.Table_213216398c104dca9ffe9a505d0e46fe [id=dd6db2a66af5463698edf91dd88d700e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.571538 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1779840e861f4e7caab371208ae15c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.574699 1166 raft_consensus.cc:2238] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.576408 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b68d23fb2df743f4afa2682f6d2e3efa (table default.Table_07f19c8f31bb48c598a7a2b8dc7187af [id=e0c7ea945a0a40e0a4a3e369c7455eca]) successfully deleted
I20250109 20:52:59.576900 1166 raft_consensus.cc:2267] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.577970 1166 ts_tablet_manager.cc:1905] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.587844 896 tablet_service.cc:1467] Processing CreateTablet for tablet b087b7546a39451089666a36d029e645 (DEFAULT_TABLE table=default.Table_1b296a6b3a0c47eb8dead9e581ac987b [id=9bda551fac0244c7bd079d717596657d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.590394 1157 ts_tablet_manager.cc:1918] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.594005 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b087b7546a39451089666a36d029e645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.594448 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4d2ad301c11943b69b59be18d6775513 (table default.Table_4f8710d177654a9c81c383e7aac0841b [id=3a1cd7e71eab4f90ae054b885cee1626]) successfully deleted
W20250109 20:52:59.595176 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b56d2c0706c84195acceb97ef4985c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b56d2c0706c84195acceb97ef4985c14 already in progress: creating tablet
I20250109 20:52:59.597072 1157 log.cc:1198] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/3d8d67b624984a0bb327d045c55476da
I20250109 20:52:59.597530 1157 ts_tablet_manager.cc:1939] T 3d8d67b624984a0bb327d045c55476da P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:52:59.599480 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9c88f42c15d94099903178218bfac1e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c88f42c15d94099903178218bfac1e5 already in progress: creating tablet
W20250109 20:52:59.599819 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 because the tablet was not found. No further retry: Not found: Tablet not found: a1ed00f7954143c9ba9bf65d8a8a68d2
I20250109 20:52:59.593686 1158 tablet_bootstrap.cc:492] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.600452 1158 ts_tablet_manager.cc:1397] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.002s
I20250109 20:52:59.601132 1158 raft_consensus.cc:357] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.601331 1158 raft_consensus.cc:383] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.601389 1166 ts_tablet_manager.cc:1918] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.601402 1158 raft_consensus.cc:738] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.601476 1166 log.cc:1198] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/e090f72eb1e046319b825ee5fe126332
I20250109 20:52:59.601629 1158 consensus_queue.cc:260] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.602066 1157 tablet_replica.cc:331] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.602334 1158 ts_tablet_manager.cc:1428] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:52:59.605247 1157 raft_consensus.cc:2238] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.605394 1158 tablet_bootstrap.cc:492] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.605513 1157 raft_consensus.cc:2267] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.606572 1157 ts_tablet_manager.cc:1905] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.607197 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 06c3aa20539740cba4e5d6e852ad2909 (table default.Table_5e3e729549234c39a601e29da97094d4 [id=5811deb4d7214be490d07cf674920448]) successfully deleted
I20250109 20:52:59.607426 1158 tablet_bootstrap.cc:654] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:52:59.607618 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1c5cef087cc349eba40c171b326a1f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c5cef087cc349eba40c171b326a1f8f already in progress: creating tablet
I20250109 20:52:59.610481 1160 tablet_replica.cc:331] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:52:59.610730 1160 raft_consensus.cc:2238] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.611205 1166 ts_tablet_manager.cc:1939] T e090f72eb1e046319b825ee5fe126332 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.611399 1160 raft_consensus.cc:2267] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.612066 1166 tablet_replica.cc:331] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.608904 1176 tablet_bootstrap.cc:492] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.614564 2129 raft_consensus.cc:491] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.616874 2129 raft_consensus.cc:513] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.617421 2129 leader_election.cc:290] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.616977 1166 raft_consensus.cc:2238] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.617621 1176 ts_tablet_manager.cc:1397] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.136s user 0.005s sys 0.000s
I20250109 20:52:59.617713 1166 raft_consensus.cc:2267] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.618250 1176 raft_consensus.cc:357] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.618427 1176 raft_consensus.cc:383] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.618482 1176 raft_consensus.cc:738] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.618636 1166 ts_tablet_manager.cc:1905] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.618657 1176 consensus_queue.cc:260] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.619458 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f420b7e623243e093d26c95276c5af9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.610005 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet b221e048f6f6463e84e630386c304a05 (table default.Table_43ed51a629174e3bbaad39c5d8f43ef7 [id=ce77658acdf34b7e90401b0982c045fe]) successfully deleted
W20250109 20:52:59.619947 990 leader_election.cc:343] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.620309 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1c5cef087cc349eba40c171b326a1f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c5cef087cc349eba40c171b326a1f8f already in progress: creating tablet
I20250109 20:52:59.620779 1176 ts_tablet_manager.cc:1428] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:52:59.621004 1176 tablet_bootstrap.cc:492] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.621383 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 3d8d67b624984a0bb327d045c55476da (table default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]) successfully deleted
I20250109 20:52:59.622196 1176 tablet_bootstrap.cc:654] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.622638 756 tablet_service.cc:1467] Processing CreateTablet for tablet 5b17962015e04c6b85169915586334cc (DEFAULT_TABLE table=default.Table_c0d1dadec3324eee8cc42a24ab27f3ed [id=891c9e371ac14ed6ab56b4419c45f4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.623042 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b17962015e04c6b85169915586334cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.623065 739 tablet_service.cc:1467] Processing CreateTablet for tablet 733c7059c750463aa586d9b733f1d1e9 (DEFAULT_TABLE table=default.Table_24614ad1648f41d9bc4bb77eb2b170ae [id=d452950a44b94715a2fdee7a65cba1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.623445 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733c7059c750463aa586d9b733f1d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.627719 1157 ts_tablet_manager.cc:1918] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.604995 2242 tablet.cc:1721] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:59.628837 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:59.628996 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 5b17962015e04c6b85169915586334cc (DEFAULT_TABLE table=default.Table_c0d1dadec3324eee8cc42a24ab27f3ed [id=891c9e371ac14ed6ab56b4419c45f4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.631208 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b17962015e04c6b85169915586334cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.647060 1034 tablet_service.cc:1467] Processing CreateTablet for tablet d6fa1245ccea427eb6b2c0eaadebd24e (DEFAULT_TABLE table=default.Table_7147fbe7700d4c52a663ea8345c0a31d [id=d69be15c9d4347bd957fee13dc93724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.649086 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bba303032679462c8b16f706096b6d4d
I20250109 20:52:59.650595 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:52:59.651796 1166 ts_tablet_manager.cc:1918] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.651890 1166 log.cc:1198] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/80da78917f0b4931ab1705371c1fe5dd
I20250109 20:52:59.652137 1166 ts_tablet_manager.cc:1939] T 80da78917f0b4931ab1705371c1fe5dd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.628036 758 tablet_service.cc:1467] Processing CreateTablet for tablet d6fa1245ccea427eb6b2c0eaadebd24e (DEFAULT_TABLE table=default.Table_7147fbe7700d4c52a663ea8345c0a31d [id=d69be15c9d4347bd957fee13dc93724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.652449 1157 log.cc:1198] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2ca83a49523b45be9ac7556b40746a83
I20250109 20:52:59.652676 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fa1245ccea427eb6b2c0eaadebd24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.652768 1157 ts_tablet_manager.cc:1939] T 2ca83a49523b45be9ac7556b40746a83 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.653553 1157 tablet_replica.cc:331] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.653725 1157 raft_consensus.cc:2238] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.653872 1157 raft_consensus.cc:2267] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.653702 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_045b529f170543478afcac3c58958866"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.654314 1157 ts_tablet_manager.cc:1905] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:59.654367 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_045b529f170543478afcac3c58958866 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.657027 1678 raft_consensus.cc:491] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.639722 2240 tablet.cc:1721] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:59.657159 1678 raft_consensus.cc:513] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.657276 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0019a01afbcc452a9a9374c92dc131f3
I20250109 20:52:59.657652 454 catalog_manager.cc:6596] ReplaceTablet: tablet 0019a01afbcc452a9a9374c92dc131f3 deleted and replaced by tablet 7a4b7e008a474978b91426f63dbd96ee
I20250109 20:52:59.657759 1678 leader_election.cc:290] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:59.657923 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9bebcb16a96c49088a5ac609ee22c5fb" } modify_external_catalogs: true
I20250109 20:52:59.658087 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9bebcb16a96c49088a5ac609ee22c5fb" } modify_external_catalogs: true
I20250109 20:52:59.658773 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce64e0f3a004c9387bb592cf19236e7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:59.658960 1055 raft_consensus.cc:2463] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:59.659310 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fa1245ccea427eb6b2c0eaadebd24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.659511 2129 raft_consensus.cc:491] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.659633 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:59.659622 2129 raft_consensus.cc:513] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.659842 713 leader_election.cc:304] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:59.660121 2129 leader_election.cc:290] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.660600 1984 raft_consensus.cc:2798] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:52:59.660728 1984 raft_consensus.cc:491] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:59.660815 1984 raft_consensus.cc:3054] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:59.660995 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba303032679462c8b16f706096b6d4d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:59.662837 990 leader_election.cc:343] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.663056 1984 raft_consensus.cc:513] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.664744 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce64e0f3a004c9387bb592cf19236e7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:59.664845 1984 leader_election.cc:290] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:59.664934 1055 raft_consensus.cc:3054] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:59.667165 1055 raft_consensus.cc:2463] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:59.667964 451 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:52:59.668244 451 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 9c3af2a00ab04b41915cffc3eb9206bc (table default.Table_14c64344192b45778141de193a1c433d [id=ffee48c2614543b981c241ea4a3dc24e]). Expected version 2 got 0
I20250109 20:52:59.668567 713 leader_election.cc:304] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:59.669207 1984 raft_consensus.cc:2798] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:59.669379 1984 raft_consensus.cc:695] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:59.669724 1984 consensus_queue.cc:237] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.670660 1158 tablet_bootstrap.cc:492] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.670842 1158 ts_tablet_manager.cc:1397] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.001s
I20250109 20:52:59.671551 1158 raft_consensus.cc:357] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.671798 1158 raft_consensus.cc:383] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.678900 2129 raft_consensus.cc:491] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.679018 2129 raft_consensus.cc:513] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.679524 2129 leader_election.cc:290] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.680451 1166 tablet_replica.cc:331] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.680706 1166 raft_consensus.cc:2238] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.680415 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4160bb4ba40347bda48f8fa88f8b6001" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.681818 1158 raft_consensus.cc:738] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.682111 1158 consensus_queue.cc:260] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.682631 1158 ts_tablet_manager.cc:1428] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:52:59.682924 1158 tablet_bootstrap.cc:492] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.683346 1166 raft_consensus.cc:2267] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:52:59.681720 990 leader_election.cc:343] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.681298 1176 tablet_bootstrap.cc:492] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.683621 1176 ts_tablet_manager.cc:1397] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250109 20:52:59.684355 1176 raft_consensus.cc:357] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.684513 1176 raft_consensus.cc:383] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.684568 1176 raft_consensus.cc:738] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.684800 1176 consensus_queue.cc:260] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.684995 1158 tablet_bootstrap.cc:654] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.615286 2241 tablet.cc:1721] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:52:59.617844 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f420b7e623243e093d26c95276c5af9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.691403 1160 ts_tablet_manager.cc:1905] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.691507 1176 ts_tablet_manager.cc:1428] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250109 20:52:59.658736 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce64e0f3a004c9387bb592cf19236e7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.691756 1176 tablet_bootstrap.cc:492] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:52:59.691936 989 leader_election.cc:343] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.692034 989 leader_election.cc:304] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.692409 2129 raft_consensus.cc:2743] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.663743 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba303032679462c8b16f706096b6d4d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.664472 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce64e0f3a004c9387bb592cf19236e7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:52:59.680135 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4160bb4ba40347bda48f8fa88f8b6001" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.693500 1176 tablet_bootstrap.cc:654] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.682684 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4b7e008a474978b91426f63dbd96ee) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:59.697408 1166 ts_tablet_manager.cc:1905] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:52:59.699873 712 leader_election.cc:343] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.700268 712 leader_election.cc:343] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.701028 989 leader_election.cc:343] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.701150 989 leader_election.cc:304] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:59.701551 989 leader_election.cc:343] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.701637 989 leader_election.cc:304] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.702051 2129 raft_consensus.cc:2743] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.702227 2129 raft_consensus.cc:2743] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.709702 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:59.714568 1157 ts_tablet_manager.cc:1918] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.716595 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c2ad7ac382e141cdbf0e04072bc26ffe" } new_table_name: "default.Table_4033dea6a18347b4b0eaf9022ceb6402" modify_external_catalogs: true
I20250109 20:52:59.718235 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 46df992009994d9e879109ba14bb06c0 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:59.719314 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d5db66989ced4d6fb84352f19e087cc7" } modify_external_catalogs: true
I20250109 20:52:59.719478 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d5db66989ced4d6fb84352f19e087cc7" } modify_external_catalogs: true
I20250109 20:52:59.720589 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b806214625f649cfa01e6ed30915e3dc" } new_table_name: "default.Table_9fe0192ffee74ba7a8398e7737878987" modify_external_catalogs: true
I20250109 20:52:59.722875 1157 log.cc:1198] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/0f502d666aa043e2a0b67a4e21bcf03e
I20250109 20:52:59.723243 1157 ts_tablet_manager.cc:1939] T 0f502d666aa043e2a0b67a4e21bcf03e P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.728089 1157 tablet_replica.cc:331] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.728499 1157 raft_consensus.cc:2238] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.729040 1157 raft_consensus.cc:2267] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.730384 1157 ts_tablet_manager.cc:1905] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.731702 1166 ts_tablet_manager.cc:1918] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.734659 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 733c7059c750463aa586d9b733f1d1e9 (DEFAULT_TABLE table=default.Table_24614ad1648f41d9bc4bb77eb2b170ae [id=d452950a44b94715a2fdee7a65cba1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.735116 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733c7059c750463aa586d9b733f1d1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.735152 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 88cd7f9808d547ca80a76e4b9568cd0a (DEFAULT_TABLE table=default.Table_9db2524d4be648e4828df22a952c1cb4 [id=f60e6d2c78664f3da51dc6cf6a1a9d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.735455 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88cd7f9808d547ca80a76e4b9568cd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.735690 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 94ed3ab88fd8470bbbc27d801323dc9d (DEFAULT_TABLE table=default.Table_30b1ffb8aacc48c6883ed04b539160e7 [id=eb5ba91d8d3a4f959e2b85bb77a154cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.736097 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ed3ab88fd8470bbbc27d801323dc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.740231 1166 log.cc:1198] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/18f85944f66543fe8dd634ad0da0728d
I20250109 20:52:59.734926 758 tablet_service.cc:1467] Processing CreateTablet for tablet 88cd7f9808d547ca80a76e4b9568cd0a (DEFAULT_TABLE table=default.Table_9db2524d4be648e4828df22a952c1cb4 [id=f60e6d2c78664f3da51dc6cf6a1a9d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.740780 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88cd7f9808d547ca80a76e4b9568cd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.741026 1166 ts_tablet_manager.cc:1939] T 18f85944f66543fe8dd634ad0da0728d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.741911 1166 tablet_replica.cc:331] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.742600 755 tablet_service.cc:1467] Processing CreateTablet for tablet 94ed3ab88fd8470bbbc27d801323dc9d (DEFAULT_TABLE table=default.Table_30b1ffb8aacc48c6883ed04b539160e7 [id=eb5ba91d8d3a4f959e2b85bb77a154cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.743120 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ed3ab88fd8470bbbc27d801323dc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.745592 1166 raft_consensus.cc:2238] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.748909 1166 raft_consensus.cc:2267] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.749715 1166 ts_tablet_manager.cc:1905] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.757211 1157 ts_tablet_manager.cc:1918] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.758124 886 tablet_service.cc:1514] Processing DeleteTablet for tablet bba303032679462c8b16f706096b6d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:59.759680 1165 tablet_bootstrap.cc:492] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:59.759842 1165 ts_tablet_manager.cc:1397] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.435s user 0.002s sys 0.004s
I20250109 20:52:59.760550 1165 raft_consensus.cc:357] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.761878 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet e090f72eb1e046319b825ee5fe126332 (table default.Table_e9297199b4064b5f889530ac2757c3ad [id=53929cac72e44474a2dbc7ae26524da3]) successfully deleted
I20250109 20:52:59.762415 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 80da78917f0b4931ab1705371c1fe5dd (table default.Table_500eb283bc7f4131b62740a5e293b785 [id=16cb6983d685473c8d7aba4e39573aae]) successfully deleted
W20250109 20:52:59.763846 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9c88f42c15d94099903178218bfac1e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c88f42c15d94099903178218bfac1e5 already in progress: creating tablet
I20250109 20:52:59.763911 1026 tablet_service.cc:1467] Processing CreateTablet for tablet dcba62cbadf04761822bf7161c52b758 (DEFAULT_TABLE table=default.Table_ea166542339a4980aea4afc8f128edd5 [id=2172881080574ac683ba26588c37cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.764396 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcba62cbadf04761822bf7161c52b758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.765458 1176 tablet_bootstrap.cc:492] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.765614 1176 ts_tablet_manager.cc:1397] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.074s user 0.005s sys 0.005s
I20250109 20:52:59.765761 2250 raft_consensus.cc:491] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.765940 2250 raft_consensus.cc:513] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.766319 1176 raft_consensus.cc:357] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.766453 1176 raft_consensus.cc:383] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.766489 1176 raft_consensus.cc:738] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.766592 2250 leader_election.cc:290] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.766673 1176 consensus_queue.cc:260] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.773978 1176 ts_tablet_manager.cc:1428] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250109 20:52:59.780920 2250 raft_consensus.cc:491] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.781018 2250 raft_consensus.cc:513] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.781500 2250 leader_election.cc:290] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.782124 1176 tablet_bootstrap.cc:492] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.788033 1166 ts_tablet_manager.cc:1918] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.766978 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a9aba1c7b424db21c30b2e92323c9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.777453 1158 tablet_bootstrap.cc:492] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.782135 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3af2a00ab04b41915cffc3eb9206bc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.788231 1157 log.cc:1198] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/aa7d7695ff5946abb6ee576431d2e87f
I20250109 20:52:59.788976 1158 ts_tablet_manager.cc:1397] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.000s
I20250109 20:52:59.793421 1157 ts_tablet_manager.cc:1939] T aa7d7695ff5946abb6ee576431d2e87f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.793953 1157 tablet_replica.cc:331] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:52:59.793922 1158 raft_consensus.cc:357] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.794131 1158 raft_consensus.cc:383] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.794203 1158 raft_consensus.cc:738] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:52:59.788555 990 leader_election.cc:343] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.794428 990 leader_election.cc:343] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.794219 1176 tablet_bootstrap.cc:654] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.794407 1158 consensus_queue.cc:260] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.796396 1166 log.cc:1198] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/8f420b7e623243e093d26c95276c5af9
I20250109 20:52:59.796684 1166 ts_tablet_manager.cc:1939] T 8f420b7e623243e093d26c95276c5af9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.798020 1166 tablet_replica.cc:331] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.798236 1166 raft_consensus.cc:2238] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.799849 1166 raft_consensus.cc:2267] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.800477 1166 ts_tablet_manager.cc:1905] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.809975 1176 tablet_bootstrap.cc:492] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.810153 1176 ts_tablet_manager.cc:1397] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250109 20:52:59.810886 1176 raft_consensus.cc:357] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.811146 1176 raft_consensus.cc:383] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.811272 1176 raft_consensus.cc:738] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.784978 2251 raft_consensus.cc:491] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.811522 1176 consensus_queue.cc:260] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.811720 2251 raft_consensus.cc:513] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.812414 2251 leader_election.cc:290] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:59.814801 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1891cbef8f4b0cae0c15d478db4860" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:52:59.814965 1055 raft_consensus.cc:2463] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:52:59.767156 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a9aba1c7b424db21c30b2e92323c9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.815979 1165 raft_consensus.cc:383] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.816161 1165 raft_consensus.cc:738] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.812738 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1891cbef8f4b0cae0c15d478db4860" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.816440 1165 consensus_queue.cc:260] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:59.816870 989 leader_election.cc:343] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.816984 989 leader_election.cc:304] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:52:59.817231 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 53379.3us
I20250109 20:52:59.817562 2250 raft_consensus.cc:2743] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.817698 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 18f85944f66543fe8dd634ad0da0728d (table default.Table_9cbf7236e62e4c22a28e6ccb34f76495 [id=f3bf80bcfbb24031a4050f6f53569e93]) successfully deleted
I20250109 20:52:59.818320 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 8f420b7e623243e093d26c95276c5af9 (table default.Table_bb47b9b8617f4706aeb895a16ded22ee [id=ec17429d6df548118b1e53e77bac80c0]) successfully deleted
W20250109 20:52:59.818544 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1c5cef087cc349eba40c171b326a1f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c5cef087cc349eba40c171b326a1f8f already in progress: creating tablet
I20250109 20:52:59.819092 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet ea477dc845614ea78e71e45f862b57ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.767524 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2ff361cf1b484434b63f79b7ae7da8a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.819298 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet bba303032679462c8b16f706096b6d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.770473 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e3a5454d3c5f44f09a050ae3d20800ae" } modify_external_catalogs: true
I20250109 20:52:59.819628 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e3a5454d3c5f44f09a050ae3d20800ae" } modify_external_catalogs: true
I20250109 20:52:59.772375 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 81db8c605d8944c89d9184b81de8e5d6 from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:59.772753 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a] alter complete (version 2)
W20250109 20:52:59.777161 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c8d8d771b56407d94fcb23d76796039 consensus state has no leader
I20250109 20:52:59.820556 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bfad8a48292647e68716ec5438e5a6a7" } new_table_name: "default.Table_702ee9d09df94d92bbd7718ccdb1a590" modify_external_catalogs: true
W20250109 20:52:59.777442 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0860554cac64b8882e39fa62462b1aa consensus state has no leader
I20250109 20:52:59.821950 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5e53bf0c9fce4d04a077919ffb080698" } new_table_name: "default.Table_6859465fc7f4440eaf4c387bc97b02ad" modify_external_catalogs: true
I20250109 20:52:59.822165 1176 ts_tablet_manager.cc:1428] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:52:59.823505 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bd1103b105f94221b1bed1edfe6477e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.824211 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd1103b105f94221b1bed1edfe6477e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:52:59.826566 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ff361cf1b484434b63f79b7ae7da8a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.781915 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3af2a00ab04b41915cffc3eb9206bc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.831434 1165 ts_tablet_manager.cc:1428] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.071s user 0.001s sys 0.001s
W20250109 20:52:59.832428 989 leader_election.cc:343] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.832533 989 leader_election.cc:304] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.832906 1176 tablet_bootstrap.cc:492] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.813045 1158 ts_tablet_manager.cc:1428] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.020s user 0.002s sys 0.002s
I20250109 20:52:59.794207 1157 raft_consensus.cc:2238] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:52:59.834290 1157 raft_consensus.cc:2267] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.834345 1158 tablet_bootstrap.cc:492] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.834793 1176 tablet_bootstrap.cc:654] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.834795 1157 ts_tablet_manager.cc:1905] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.819307 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4b7e008a474978b91426f63dbd96ee) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.836181 1158 tablet_bootstrap.cc:654] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.836863 2250 raft_consensus.cc:2743] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.814077 758 tablet_service.cc:1467] Processing CreateTablet for tablet dcba62cbadf04761822bf7161c52b758 (DEFAULT_TABLE table=default.Table_ea166542339a4980aea4afc8f128edd5 [id=2172881080574ac683ba26588c37cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.815845 713 leader_election.cc:304] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:52:59.828572 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4b7e008a474978b91426f63dbd96ee) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.828684 739 tablet_service.cc:1514] Processing DeleteTablet for tablet bba303032679462c8b16f706096b6d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.840636 2251 raft_consensus.cc:2798] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250109 20:52:59.840795 712 leader_election.cc:343] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.848526 1165 tablet_bootstrap.cc:492] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:59.864984 1165 tablet_bootstrap.cc:654] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.865571 2251 raft_consensus.cc:491] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:52:59.866189 2251 raft_consensus.cc:3054] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:59.870258 1176 tablet_bootstrap.cc:492] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.870451 1176 ts_tablet_manager.cc:1397] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20250109 20:52:59.871186 1176 raft_consensus.cc:357] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.871549 1176 raft_consensus.cc:383] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.871630 1176 raft_consensus.cc:738] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.871842 1176 consensus_queue.cc:260] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.876672 1176 ts_tablet_manager.cc:1428] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:52:59.877022 1176 tablet_bootstrap.cc:492] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.882393 1176 tablet_bootstrap.cc:654] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.886632 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2ca83a49523b45be9ac7556b40746a83 (table default.Table_79f5ff4b60444c1fb9e227b39c89e4f4 [id=523cbdd3d4344d24afca6003e115d5ed]) successfully deleted
I20250109 20:52:59.887115 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 0f502d666aa043e2a0b67a4e21bcf03e (table default.Table_b086cd04a8e249a09d729be46eda67b1 [id=6aa6003be0684dc88b8ed001eed73727]) successfully deleted
I20250109 20:52:59.887521 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet aa7d7695ff5946abb6ee576431d2e87f (table default.Table_d311b482665944949ba9d26a91d7fd39 [id=d1eb369ad1da4221b0be6419fe55ad33]) successfully deleted
I20250109 20:52:59.873487 1158 tablet_bootstrap.cc:492] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.888878 1158 ts_tablet_manager.cc:1397] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.000s
I20250109 20:52:59.889595 1158 raft_consensus.cc:357] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.889786 1158 raft_consensus.cc:383] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.889875 1158 raft_consensus.cc:738] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.890106 1158 consensus_queue.cc:260] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.890578 1158 ts_tablet_manager.cc:1428] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250109 20:52:59.892930 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet beee9825fa0240e88029bf32ace0e187 consensus state has no leader
I20250109 20:52:59.890374 1176 tablet_bootstrap.cc:492] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:52:59.893774 1176 ts_tablet_manager.cc:1397] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250109 20:52:59.894454 1176 raft_consensus.cc:357] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.894621 1176 raft_consensus.cc:383] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.894662 1176 raft_consensus.cc:738] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.894863 1176 consensus_queue.cc:260] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.894981 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 1c65489a602141239c99e8e8dae22ab4 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]). Expected version 4 got 0
I20250109 20:52:59.896101 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ab28e2a04600484e858415ada1017c00" } modify_external_catalogs: true
I20250109 20:52:59.896263 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ab28e2a04600484e858415ada1017c00" } modify_external_catalogs: true
I20250109 20:52:59.896567 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcba62cbadf04761822bf7161c52b758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.902397 1166 ts_tablet_manager.cc:1918] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:52:59.902580 1166 log.cc:1198] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4160bb4ba40347bda48f8fa88f8b6001
I20250109 20:52:59.902916 1166 ts_tablet_manager.cc:1939] T 4160bb4ba40347bda48f8fa88f8b6001 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:52:59.903203 2251 raft_consensus.cc:513] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.904151 2251 leader_election.cc:290] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:52:59.905102 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1891cbef8f4b0cae0c15d478db4860" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:52:59.907125 712 leader_election.cc:343] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.910638 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1891cbef8f4b0cae0c15d478db4860" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:52:59.910809 1052 raft_consensus.cc:3054] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:52:59.916060 1165 tablet_bootstrap.cc:492] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:59.916245 1165 ts_tablet_manager.cc:1397] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.003s
I20250109 20:52:59.916894 1165 raft_consensus.cc:357] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.917117 1165 raft_consensus.cc:383] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.917189 1165 raft_consensus.cc:738] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.917411 1165 consensus_queue.cc:260] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:52:59.918044 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f407b76780974cf7b3635fa1bf1585ec consensus state has no leader
I20250109 20:52:59.919982 453 catalog_manager.cc:5526] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:52:59.920485 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]). Expected version 3 got 0
I20250109 20:52:59.921268 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:52:59.921382 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8a70d90c9fb04010bfc9fb5342c82862"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.921581 1165 ts_tablet_manager.cc:1428] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20250109 20:52:59.921870 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a70d90c9fb04010bfc9fb5342c82862 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.924443 1165 tablet_bootstrap.cc:492] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:52:59.926484 1165 tablet_bootstrap.cc:654] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.926712 2129 raft_consensus.cc:491] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.926853 2129 raft_consensus.cc:513] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.927435 2129 leader_election.cc:290] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.928378 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165f6304acc849569a8e90d135a8e2d0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:52:59.928576 1176 ts_tablet_manager.cc:1428] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.035s user 0.001s sys 0.000s
I20250109 20:52:59.928822 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165f6304acc849569a8e90d135a8e2d0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:52:59.929111 990 leader_election.cc:343] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:52:59.929420 989 leader_election.cc:343] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.929505 989 leader_election.cc:304] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.929911 2129 raft_consensus.cc:2743] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:52:59.931198 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:52:59.933413 886 tablet_service.cc:1467] Processing CreateTablet for tablet d30d928c44a84263ad93d9ee36b2d632 (DEFAULT_TABLE table=default.Table_c075a9e071d84c4ca107998bcdc8af22 [id=bb6d0b755dd94e3793aeaea3f1099648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.933775 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30d928c44a84263ad93d9ee36b2d632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.934499 896 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7e3652017946d481090b9940d32e97 (DEFAULT_TABLE table=default.Table_d1a9999752c4415ca70dcb514d0e0003 [id=b160b9bc7e574e1896000b46a6114ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.934751 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7e3652017946d481090b9940d32e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.937258 885 tablet_service.cc:1467] Processing CreateTablet for tablet f2a9d7750a384b408d6691176707e38b (DEFAULT_TABLE table=default.Table_6c856ba75072493c881fa3bb24292947 [id=51b5c80901184c929db681938cc074c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.937505 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a9d7750a384b408d6691176707e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.938834 455 master_service.cc:946] ReplaceTablet: received request to replace tablet cca642fb9c3e4a55adfb6ad8386a67cd from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:59.939567 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 9d1c9f392439440bba88a86dbd6d853e from {username='slave'} at 127.0.0.1:33768
I20250109 20:52:59.941105 1165 tablet_bootstrap.cc:492] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:52:59.941272 1165 ts_tablet_manager.cc:1397] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250109 20:52:59.942368 1176 tablet_bootstrap.cc:492] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:52:59.943742 1176 tablet_bootstrap.cc:654] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.943774 1166 tablet_replica.cc:331] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:52:59.943944 1166 raft_consensus.cc:2238] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:52:59.944195 1166 raft_consensus.cc:2267] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:52:59.944595 1166 ts_tablet_manager.cc:1905] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.946198 1158 tablet_bootstrap.cc:492] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.947425 1158 tablet_bootstrap.cc:654] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.949803 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 006475691733470f8a0468365972dfd2
I20250109 20:52:59.956812 739 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7e3652017946d481090b9940d32e97 (DEFAULT_TABLE table=default.Table_d1a9999752c4415ca70dcb514d0e0003 [id=b160b9bc7e574e1896000b46a6114ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.957206 756 tablet_service.cc:1467] Processing CreateTablet for tablet d30d928c44a84263ad93d9ee36b2d632 (DEFAULT_TABLE table=default.Table_c075a9e071d84c4ca107998bcdc8af22 [id=bb6d0b755dd94e3793aeaea3f1099648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.957914 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:52:59.957968 755 tablet_service.cc:1467] Processing CreateTablet for tablet f2a9d7750a384b408d6691176707e38b (DEFAULT_TABLE table=default.Table_6c856ba75072493c881fa3bb24292947 [id=51b5c80901184c929db681938cc074c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.959424 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2975f30fccea4d25b3001a0b5a91d8d4
I20250109 20:52:59.961130 1017 tablet_service.cc:1467] Processing CreateTablet for tablet d30d928c44a84263ad93d9ee36b2d632 (DEFAULT_TABLE table=default.Table_c075a9e071d84c4ca107998bcdc8af22 [id=bb6d0b755dd94e3793aeaea3f1099648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.961467 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30d928c44a84263ad93d9ee36b2d632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.961546 746 tablet_service.cc:1467] Processing CreateTablet for tablet b3d7dfc1a7dd46898ba7a463ce3f12b8 (DEFAULT_TABLE table=default.Table_31e395d70eeb405c97b2221c23dd329a [id=fdbd95007ca14da49a657221fd773544]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.962553 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f9b7d3b2a65544bf894ec96e183e2d8c" } modify_external_catalogs: true
I20250109 20:52:59.962708 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f9b7d3b2a65544bf894ec96e183e2d8c" } modify_external_catalogs: true
I20250109 20:52:59.965679 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a95a31468b431fb4304cf6680016da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.965898 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 225046d45c1c43b488babf5da8de75d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.965898 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 8539cefebb844b3098a308ca580b0e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:52:59.966241 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f2a9d7750a384b408d6691176707e38b (DEFAULT_TABLE table=default.Table_6c856ba75072493c881fa3bb24292947 [id=51b5c80901184c929db681938cc074c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.966323 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 46df992009994d9e879109ba14bb06c0
I20250109 20:52:59.966506 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a9d7750a384b408d6691176707e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.969627 1158 tablet_bootstrap.cc:492] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.969813 1158 ts_tablet_manager.cc:1397] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20250109 20:52:59.970765 745 tablet_service.cc:1467] Processing CreateTablet for tablet 714ef0fa6b0d4ba9b03383baec160cc8 (DEFAULT_TABLE table=default.Table_f9b7d3b2a65544bf894ec96e183e2d8c [id=c259df784a334de6aa2c7e7e782f7807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.971169 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 714ef0fa6b0d4ba9b03383baec160cc8 (DEFAULT_TABLE table=default.Table_f9b7d3b2a65544bf894ec96e183e2d8c [id=c259df784a334de6aa2c7e7e782f7807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.970526 1158 raft_consensus.cc:357] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.971587 753 tablet_service.cc:1467] Processing CreateTablet for tablet 74690c11b790472c90801bdf57c7cc1c (DEFAULT_TABLE table=default.Table_43ac00c1a42746499f265e3beafcc8c3 [id=53eb787cd49b4c97aa7026b61c11ae94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.971657 1158 raft_consensus.cc:383] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:52:59.972314 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 74690c11b790472c90801bdf57c7cc1c (DEFAULT_TABLE table=default.Table_43ac00c1a42746499f265e3beafcc8c3 [id=53eb787cd49b4c97aa7026b61c11ae94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.972432 740 tablet_service.cc:1467] Processing CreateTablet for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 (DEFAULT_TABLE table=default.Table_7fb50e2939fb4fe39c63669d58697188 [id=3bb3cd4a3a60415fbb6e5f23eaecd7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.973023 2129 raft_consensus.cc:491] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:52:59.973132 2129 raft_consensus.cc:513] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.973382 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 81db8c605d8944c89d9184b81de8e5d6
I20250109 20:52:59.973660 2129 leader_election.cc:290] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:52:59.973649 1158 raft_consensus.cc:738] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.973884 1158 consensus_queue.cc:260] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.974092 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea477dc845614ea78e71e45f862b57ff" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:52:59.974524 990 leader_election.cc:343] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:52:59.974617 1158 ts_tablet_manager.cc:1428] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:52:59.974961 1158 tablet_bootstrap.cc:492] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:52:59.975131 747 tablet_service.cc:1467] Processing CreateTablet for tablet bd799b72a7f445089655304d561c299d (DEFAULT_TABLE table=default.Table_696e8a1e0e3145bab03b3d71484d37cc [id=aa49cd448f664e67af4075b75df4237c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.976591 1158 tablet_bootstrap.cc:654] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:52:59.977607 1019 tablet_service.cc:1467] Processing CreateTablet for tablet bd799b72a7f445089655304d561c299d (DEFAULT_TABLE table=default.Table_696e8a1e0e3145bab03b3d71484d37cc [id=aa49cd448f664e67af4075b75df4237c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.977919 1023 tablet_service.cc:1467] Processing CreateTablet for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 (DEFAULT_TABLE table=default.Table_7fb50e2939fb4fe39c63669d58697188 [id=3bb3cd4a3a60415fbb6e5f23eaecd7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.978291 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b3d7dfc1a7dd46898ba7a463ce3f12b8 (DEFAULT_TABLE table=default.Table_31e395d70eeb405c97b2221c23dd329a [id=fdbd95007ca14da49a657221fd773544]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.978893 446 catalog_manager.cc:6596] ReplaceTablet: tablet 46df992009994d9e879109ba14bb06c0 deleted and replaced by tablet d11e17f3391740b79ffa9d8a8c9bf400
I20250109 20:52:59.979131 2240 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:59.979976 447 catalog_manager.cc:6596] ReplaceTablet: tablet 81db8c605d8944c89d9184b81de8e5d6 deleted and replaced by tablet cfa50adc9f7449bb8b5eb3243b5f9405
I20250109 20:52:59.980508 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:52:59.981598 750 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1a3d64ba894908a72807038c7d1d92 (DEFAULT_TABLE table=default.Table_5af25870e6cc4806873d8324713da6c1 [id=d24d737426484227952c75e72f68bca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.982965 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1a3d64ba894908a72807038c7d1d92 (DEFAULT_TABLE table=default.Table_5af25870e6cc4806873d8324713da6c1 [id=d24d737426484227952c75e72f68bca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.983125 742 tablet_service.cc:1467] Processing CreateTablet for tablet 321489e81e324d2586ce2b8ff82c01a0 (DEFAULT_TABLE table=default.Table_ff19ec0caa6049ddbedbf619756c83bd [id=cc9f1fd810ec456d80bb6f74ba970b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.983361 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 321489e81e324d2586ce2b8ff82c01a0 (DEFAULT_TABLE table=default.Table_ff19ec0caa6049ddbedbf619756c83bd [id=cc9f1fd810ec456d80bb6f74ba970b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.983551 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2ca3accb8f0349d8bdd2a2306a9b1423"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.984052 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ca3accb8f0349d8bdd2a2306a9b1423 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.984406 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 (DEFAULT_TABLE table=default.Table_6ba9903051db4a88adae464061f959d8 [id=1caa071b594241f1b6ec7bd658558347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.984900 754 tablet_service.cc:1467] Processing CreateTablet for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 (DEFAULT_TABLE table=default.Table_6ba9903051db4a88adae464061f959d8 [id=1caa071b594241f1b6ec7bd658558347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.985462 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4784d2e8394f4928a758483708ee3504"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.985926 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4784d2e8394f4928a758483708ee3504 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.987068 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c75db1b93cbd4edb9fce068615c63752"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.987469 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c75db1b93cbd4edb9fce068615c63752 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.988530 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b6729d452f65409992e893d661ace4db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:52:59.989123 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6729d452f65409992e893d661ace4db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.989610 752 tablet_service.cc:1467] Processing CreateTablet for tablet 31af7fe649b94a44a1fdf71bdd5fd083 (DEFAULT_TABLE table=default.Table_7768e94e2bff455eb490b7cd4e6f1fbb [id=e4d720450c6141fb82420e40b69c13ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.989887 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 31af7fe649b94a44a1fdf71bdd5fd083 (DEFAULT_TABLE table=default.Table_7768e94e2bff455eb490b7cd4e6f1fbb [id=e4d720450c6141fb82420e40b69c13ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.990654 751 tablet_service.cc:1467] Processing CreateTablet for tablet dba4d7995a414f43800e797ceb01ee66 (DEFAULT_TABLE table=default.Table_48fcb713e28c46928c9308b50268e442 [id=2b9e7d2c366349cbb8359b670a068b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.991112 1018 tablet_service.cc:1467] Processing CreateTablet for tablet dba4d7995a414f43800e797ceb01ee66 (DEFAULT_TABLE table=default.Table_48fcb713e28c46928c9308b50268e442 [id=2b9e7d2c366349cbb8359b670a068b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.991197 757 tablet_service.cc:1467] Processing CreateTablet for tablet 20d27588609f465c8dedae173e57e9b2 (DEFAULT_TABLE table=default.Table_d7b0c3e0c004488b982f368831954313 [id=2c8afc281d7c46cab5824b64446e9403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.990345 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 0f7e3652017946d481090b9940d32e97 (DEFAULT_TABLE table=default.Table_d1a9999752c4415ca70dcb514d0e0003 [id=b160b9bc7e574e1896000b46a6114ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.992098 1052 raft_consensus.cc:2463] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:52:59.992210 1157 ts_tablet_manager.cc:1918] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:52:59.992292 1157 log.cc:1198] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f672cbd6dffc49dbb69992ae7db979ea
I20250109 20:52:59.992599 1157 ts_tablet_manager.cc:1939] T f672cbd6dffc49dbb69992ae7db979ea P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:52:59.992807 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9af5cc3af3424aa3bd127450d55d6b36
I20250109 20:52:59.992859 713 leader_election.cc:304] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:52:59.993176 2251 raft_consensus.cc:2798] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:52:59.993264 2251 raft_consensus.cc:695] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:52:59.993438 2251 consensus_queue.cc:237] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.994633 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1a3d64ba894908a72807038c7d1d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.994920 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba4d7995a414f43800e797ceb01ee66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.995510 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ad55b91a6c405aae3ed456a8c4d9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.995605 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74690c11b790472c90801bdf57c7cc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.995692 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb880d4da1ca4c2c851fc8c1e0846ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.995906 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 714ef0fa6b0d4ba9b03383baec160cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.996045 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba4d7995a414f43800e797ceb01ee66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.996176 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd799b72a7f445089655304d561c299d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.996301 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 714ef0fa6b0d4ba9b03383baec160cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.996403 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321489e81e324d2586ce2b8ff82c01a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.996506 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d27588609f465c8dedae173e57e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.996529 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d7dfc1a7dd46898ba7a463ce3f12b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.996903 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb880d4da1ca4c2c851fc8c1e0846ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.996942 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74690c11b790472c90801bdf57c7cc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.997295 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7e3652017946d481090b9940d32e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.997357 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31af7fe649b94a44a1fdf71bdd5fd083. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.997591 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30d928c44a84263ad93d9ee36b2d632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:52:59.997484 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_75546481297c4d7f9038188240eb1852"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:52:59.997784 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1a3d64ba894908a72807038c7d1d92. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:52:59.997931 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75546481297c4d7f9038188240eb1852 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.998131 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd799b72a7f445089655304d561c299d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.003078 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321489e81e324d2586ce2b8ff82c01a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.003408 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ad55b91a6c405aae3ed456a8c4d9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.003713 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31af7fe649b94a44a1fdf71bdd5fd083. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.005573 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f7e3652017946d481090b9940d32e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.009692 1157 tablet_replica.cc:331] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.009878 1157 raft_consensus.cc:2238] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.010004 1157 raft_consensus.cc:2267] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.010459 1157 ts_tablet_manager.cc:1905] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:52:59.941995 1165 raft_consensus.cc:357] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:52:59.945574 893 tablet_service.cc:1467] Processing CreateTablet for tablet 714ef0fa6b0d4ba9b03383baec160cc8 (DEFAULT_TABLE table=default.Table_f9b7d3b2a65544bf894ec96e183e2d8c [id=c259df784a334de6aa2c7e7e782f7807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.946151 890 tablet_service.cc:1467] Processing CreateTablet for tablet 74690c11b790472c90801bdf57c7cc1c (DEFAULT_TABLE table=default.Table_43ac00c1a42746499f265e3beafcc8c3 [id=53eb787cd49b4c97aa7026b61c11ae94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.946624 880 tablet_service.cc:1467] Processing CreateTablet for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 (DEFAULT_TABLE table=default.Table_7fb50e2939fb4fe39c63669d58697188 [id=3bb3cd4a3a60415fbb6e5f23eaecd7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.947116 889 tablet_service.cc:1467] Processing CreateTablet for tablet bd799b72a7f445089655304d561c299d (DEFAULT_TABLE table=default.Table_696e8a1e0e3145bab03b3d71484d37cc [id=aa49cd448f664e67af4075b75df4237c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.947580 894 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1a3d64ba894908a72807038c7d1d92 (DEFAULT_TABLE table=default.Table_5af25870e6cc4806873d8324713da6c1 [id=d24d737426484227952c75e72f68bca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.948045 882 tablet_service.cc:1467] Processing CreateTablet for tablet 321489e81e324d2586ce2b8ff82c01a0 (DEFAULT_TABLE table=default.Table_ff19ec0caa6049ddbedbf619756c83bd [id=cc9f1fd810ec456d80bb6f74ba970b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.949105 883 tablet_service.cc:1467] Processing CreateTablet for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 (DEFAULT_TABLE table=default.Table_6ba9903051db4a88adae464061f959d8 [id=1caa071b594241f1b6ec7bd658558347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.016891 1158 tablet_bootstrap.cc:492] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.949607 887 tablet_service.cc:1467] Processing CreateTablet for tablet 31af7fe649b94a44a1fdf71bdd5fd083 (DEFAULT_TABLE table=default.Table_7768e94e2bff455eb490b7cd4e6f1fbb [id=e4d720450c6141fb82420e40b69c13ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.016924 1165 raft_consensus.cc:383] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.016999 1158 ts_tablet_manager.cc:1397] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250109 20:53:00.016991 1165 raft_consensus.cc:738] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:52:59.950106 897 tablet_service.cc:1467] Processing CreateTablet for tablet dba4d7995a414f43800e797ceb01ee66 (DEFAULT_TABLE table=default.Table_48fcb713e28c46928c9308b50268e442 [id=2b9e7d2c366349cbb8359b670a068b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.950587 891 tablet_service.cc:1467] Processing CreateTablet for tablet 20d27588609f465c8dedae173e57e9b2 (DEFAULT_TABLE table=default.Table_d7b0c3e0c004488b982f368831954313 [id=2c8afc281d7c46cab5824b64446e9403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.950779 899 tablet_service.cc:1467] Processing CreateTablet for tablet de57e94d00d748dc8b02d60f2f1fb27a (DEFAULT_TABLE table=default.Table_dc20220cc040480385a78533a6fe34e8 [id=6a133a1babbe4a4093b02bbecb1f7e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.951257 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7357b4f67dc6448dbb8f77dce81731f3 (DEFAULT_TABLE table=default.Table_5b016d79eedf43bbbcdc6d742dde478e [id=e030b9f57454438984429bebb8aff4a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.951745 884 tablet_service.cc:1467] Processing CreateTablet for tablet 362b83f6d6c84fd48a3d03b92e3bcf04 (DEFAULT_TABLE table=default.Table_1bc42ce5b4ac4d4283cdd4117d6fc80a [id=11b142da851f4af6b399e783d173e443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.017968 1158 raft_consensus.cc:357] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.018138 1158 raft_consensus.cc:383] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.018180 1158 raft_consensus.cc:738] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.018056 1165 consensus_queue.cc:260] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.018334 1158 consensus_queue.cc:260] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.018707 1158 ts_tablet_manager.cc:1428] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:00.018908 1158 tablet_bootstrap.cc:492] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.020342 1158 tablet_bootstrap.cc:654] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.023379 1165 ts_tablet_manager.cc:1428] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.082s user 0.001s sys 0.000s
I20250109 20:53:00.023636 1165 tablet_bootstrap.cc:492] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.025101 1165 tablet_bootstrap.cc:654] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.026873 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 08209ea49c894209abca6c0881f534f5
I20250109 20:53:00.028872 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9e7489c87b374499a6d3a5dded414b53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.029410 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e7489c87b374499a6d3a5dded414b53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.040686 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3e2af93c0f48485e93d72eb35f08e8a3" } new_table_name: "default.Table_9b677cb92a4c4926a136962b0dac08fa" modify_external_catalogs: true
I20250109 20:53:00.042934 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_86fde8a5cdef40daa1e0e95b33e93710"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.043422 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86fde8a5cdef40daa1e0e95b33e93710 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:52:59.952540 885 tablet_service.cc:1467] Processing CreateTablet for tablet b3d7dfc1a7dd46898ba7a463ce3f12b8 (DEFAULT_TABLE table=default.Table_31e395d70eeb405c97b2221c23dd329a [id=fdbd95007ca14da49a657221fd773544]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.953084 881 tablet_service.cc:1467] Processing CreateTablet for tablet d522f15b547342f4aa72cb25eb581efd (DEFAULT_TABLE table=default.Table_9ff1145eaa7f496b80139a83927b023a [id=f67740c1c16d487ab21c8bd9160dfea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.955129 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025cdef634d4626a33f01c362a1388a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:59.963371 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 2975f30fccea4d25b3001a0b5a91d8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.043962 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 006475691733470f8a0468365972dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.044270 896 tablet_service.cc:1467] Processing CreateTablet for tablet 36a3d5e666f3423db8a662d99e78a99f (DEFAULT_TABLE table=default.Table_9f92e5ee4b5b4f7fa2ec542cc8ace5f7 [id=4ecefa1b1a6f4f74ba660bd519c2a142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.044438 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9d1c9f392439440bba88a86dbd6d853e
I20250109 20:52:59.967113 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 46df992009994d9e879109ba14bb06c0 with delete_type TABLET_DATA_DELETED (replaced by tablet d11e17f3391740b79ffa9d8a8c9bf400) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.044589 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af5cc3af3424aa3bd127450d55d6b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.044745 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 08209ea49c894209abca6c0881f534f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.044898 455 catalog_manager.cc:6596] ReplaceTablet: tablet 9d1c9f392439440bba88a86dbd6d853e deleted and replaced by tablet 9c6a06112f354465912054990ba0ea58
I20250109 20:53:00.044926 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a58d439254f8418091e3f23c8c274240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.045176 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0a56e4ac9a02430886c64ffa9268293f" } new_table_name: "default.Table_45791cc3612f483986b22cd9349edda0" modify_external_catalogs: true
I20250109 20:53:00.045316 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74690c11b790472c90801bdf57c7cc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.045548 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d27588609f465c8dedae173e57e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.045723 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7357b4f67dc6448dbb8f77dce81731f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.045869 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd799b72a7f445089655304d561c299d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.046034 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321489e81e324d2586ce2b8ff82c01a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.046351 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de57e94d00d748dc8b02d60f2f1fb27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.046519 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba4d7995a414f43800e797ceb01ee66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.046506 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d53cb6e3c5514b2e89f54a956be59a83" } new_table_name: "default.Table_d8522a2e3d384ae6ad03e0449e22f3cf" modify_external_catalogs: true
I20250109 20:53:00.046670 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb880d4da1ca4c2c851fc8c1e0846ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.046836 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ad55b91a6c405aae3ed456a8c4d9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.046999 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c1a3d64ba894908a72807038c7d1d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.047513 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31af7fe649b94a44a1fdf71bdd5fd083. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.047834 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 362b83f6d6c84fd48a3d03b92e3bcf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.048138 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d522f15b547342f4aa72cb25eb581efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.048578 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a3d5e666f3423db8a662d99e78a99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.049037 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 714ef0fa6b0d4ba9b03383baec160cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.049412 1160 ts_tablet_manager.cc:1918] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:00.049500 1160 log.cc:1198] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f672cbd6dffc49dbb69992ae7db979ea
I20250109 20:53:00.049760 1165 tablet_bootstrap.cc:492] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.049929 1160 ts_tablet_manager.cc:1939] T f672cbd6dffc49dbb69992ae7db979ea P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:00.050175 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d7dfc1a7dd46898ba7a463ce3f12b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.053797 1157 ts_tablet_manager.cc:1918] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.053905 1157 log.cc:1198] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/219a4b508050490cb3f1ee418e0edbf8
I20250109 20:53:00.054148 1157 ts_tablet_manager.cc:1939] T 219a4b508050490cb3f1ee418e0edbf8 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:00.062794 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d7dfc1a7dd46898ba7a463ce3f12b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:00.071339 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e676bba315ef469da47ad841322f48e9 consensus state has no leader
W20250109 20:53:00.071775 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bf72b6386524a4d8a6a883269abc791 consensus state has no leader
W20250109 20:53:00.072460 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ad5f7eaea8c46b6ba0f83753112094e consensus state has no leader
I20250109 20:53:00.075786 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 714ef0fa6b0d4ba9b03383baec160cc8
I20250109 20:53:00.076404 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 714ef0fa6b0d4ba9b03383baec160cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.081059 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a9d7750a384b408d6691176707e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.091678 1166 ts_tablet_manager.cc:1918] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:00.091780 1166 log.cc:1198] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f672cbd6dffc49dbb69992ae7db979ea
I20250109 20:53:00.092067 1166 ts_tablet_manager.cc:1939] T f672cbd6dffc49dbb69992ae7db979ea P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:00.092451 1166 tablet_replica.cc:331] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:00.092623 1166 raft_consensus.cc:2238] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.092770 1166 raft_consensus.cc:2267] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.093206 1166 ts_tablet_manager.cc:1905] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.100275 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2972d7dcc5cd4f0e88717b5d94f99f4d" } new_table_name: "default.Table_a1280d01e103484e88adcd24c412c519" modify_external_catalogs: true
I20250109 20:53:00.100622 1165 ts_tablet_manager.cc:1397] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.000s
W20250109 20:53:00.106458 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6bcf34f4788483783acb3d09accba37 consensus state has no leader
I20250109 20:53:00.107261 1165 raft_consensus.cc:357] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.107435 1165 raft_consensus.cc:383] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.107477 1165 raft_consensus.cc:738] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.107666 1165 consensus_queue.cc:260] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.115031 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_3585fc2fc449422880a7ebe625c22d09 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 1 got 0
I20250109 20:53:00.115517 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 724d15e05a864852a118badae13a5d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.116163 2242 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:00.117059 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 362b83f6d6c84fd48a3d03b92e3bcf04 (DEFAULT_TABLE table=default.Table_1bc42ce5b4ac4d4283cdd4117d6fc80a [id=11b142da851f4af6b399e783d173e443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.117476 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 362b83f6d6c84fd48a3d03b92e3bcf04. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:00.125543 712 consensus_peers.cc:487] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:00.145151 891 tablet_service.cc:1467] Processing CreateTablet for tablet a8ad256c6bf64a699ede8ff3cee5dac1 (DEFAULT_TABLE table=default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.148965 887 tablet_service.cc:1467] Processing CreateTablet for tablet 02b5127f0b7848038e48c879a9785618 (DEFAULT_TABLE table=default.Table_5533e631d0dc4529b038c22187a08afd [id=bdd87101f968403499714582c97b09b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.149549 884 tablet_service.cc:1467] Processing CreateTablet for tablet eac75279378b4e6b8dc9256d67bd6dd4 (DEFAULT_TABLE table=default.Table_f60b81fe14f9450f8d5b0885834f3dd7 [id=1553a9cdf4ba4b41a348f8d1253053e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.149864 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eac75279378b4e6b8dc9256d67bd6dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.150925 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7357b4f67dc6448dbb8f77dce81731f3 (DEFAULT_TABLE table=default.Table_5b016d79eedf43bbbcdc6d742dde478e [id=e030b9f57454438984429bebb8aff4a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.151237 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7357b4f67dc6448dbb8f77dce81731f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.152002 919 raft_consensus.cc:3054] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:00.153528 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2975f30fccea4d25b3001a0b5a91d8d4 (DEFAULT_TABLE table=default.Table_9bebcb16a96c49088a5ac609ee22c5fb [id=ec57be40c6694016b8caa2b49931dc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.153681 1158 tablet_bootstrap.cc:492] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:00.153797 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2975f30fccea4d25b3001a0b5a91d8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.153802 1158 ts_tablet_manager.cc:1397] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.135s user 0.013s sys 0.015s
I20250109 20:53:00.154369 1158 raft_consensus.cc:357] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.154527 1158 raft_consensus.cc:383] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.154563 1158 raft_consensus.cc:738] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.154748 1158 consensus_queue.cc:260] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.155201 1158 ts_tablet_manager.cc:1428] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:00.155439 1158 tablet_bootstrap.cc:492] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.156757 1158 tablet_bootstrap.cc:654] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.156836 881 tablet_service.cc:1467] Processing CreateTablet for tablet f532f82a5696489f9240be105e03a6d5 (DEFAULT_TABLE table=default.Table_8c8845f0032e48d787d1e31838cc5c17 [id=5edb67b8f5c74744a932909df0de2be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.157117 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f532f82a5696489f9240be105e03a6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.158452 750 tablet_service.cc:1467] Processing CreateTablet for tablet d522f15b547342f4aa72cb25eb581efd (DEFAULT_TABLE table=default.Table_9ff1145eaa7f496b80139a83927b023a [id=f67740c1c16d487ab21c8bd9160dfea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.158730 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d522f15b547342f4aa72cb25eb581efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.159129 2251 consensus_queue.cc:1035] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:00.128834 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3585fc2fc449422880a7ebe625c22d09" } new_table_name: "default.Table_50d70e8a583c47059eafc658d73feb69" modify_external_catalogs: true
I20250109 20:53:00.160939 746 tablet_service.cc:1467] Processing CreateTablet for tablet de57e94d00d748dc8b02d60f2f1fb27a (DEFAULT_TABLE table=default.Table_dc20220cc040480385a78533a6fe34e8 [id=6a133a1babbe4a4093b02bbecb1f7e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.161185 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de57e94d00d748dc8b02d60f2f1fb27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.129081 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_026d505e4cfa48c69084663bbc553f83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.162163 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_026d505e4cfa48c69084663bbc553f83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.162976 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b5127f0b7848038e48c879a9785618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.132561 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_42be9454af31421f84024e4849874065"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.163372 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42be9454af31421f84024e4849874065 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.164978 754 tablet_service.cc:1467] Processing CreateTablet for tablet 362b83f6d6c84fd48a3d03b92e3bcf04 (DEFAULT_TABLE table=default.Table_1bc42ce5b4ac4d4283cdd4117d6fc80a [id=11b142da851f4af6b399e783d173e443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.165306 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 362b83f6d6c84fd48a3d03b92e3bcf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.166002 883 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5e0597d9854419a16a25d1d402174c (DEFAULT_TABLE table=default.Table_6c649042616d4c449929c794c0ba36cb [id=7c8d25a44cbe47c1a98acc4a92116495]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.166293 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5e0597d9854419a16a25d1d402174c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.170686 1158 tablet_bootstrap.cc:492] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:00.170830 1158 ts_tablet_manager.cc:1397] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20250109 20:53:00.171293 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 8ed134863bf642e79c26e5bf75b248b0 (table default.Table_6640cef9559a4ba18d06c0aed445852f [id=46c011650d964e26b40be8a854e1f8c0]). Expected version 2 got 0
I20250109 20:53:00.171468 1158 raft_consensus.cc:357] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.171627 1158 raft_consensus.cc:383] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.171675 1158 raft_consensus.cc:738] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.171917 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1c9f392439440bba88a86dbd6d853e with delete_type TABLET_DATA_DELETED (replaced by tablet 9c6a06112f354465912054990ba0ea58) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.171711 450 catalog_manager.cc:5526] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:00.171892 1158 consensus_queue.cc:260] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.172144 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 2b1891cbef8f4b0cae0c15d478db4860 (table default.Table_ca51464e265c4e28b9e54fe7be0587fc [id=0a0599668d77405ba0e72eda2aa5219e]). Expected version 4 got 0
I20250109 20:53:00.172319 1158 ts_tablet_manager.cc:1428] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250109 20:53:00.172564 1158 tablet_bootstrap.cc:492] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.174124 1158 tablet_bootstrap.cc:654] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.157337 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_915d920262ad443daadf80f5f7367d69 [id=17d12fac83d6463fb91fab6729a08139] alter complete (version 3)
I20250109 20:53:00.176699 2223 log.cc:826] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Log is configured to *not* fsync() on all Append() calls
I20250109 20:53:00.183075 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_50c2ff1060fd42b6a01b6b4f11cf3c14" } modify_external_catalogs: true
I20250109 20:53:00.183254 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_50c2ff1060fd42b6a01b6b4f11cf3c14" } modify_external_catalogs: true
I20250109 20:53:00.184396 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f60b81fe14f9450f8d5b0885834f3dd7" } modify_external_catalogs: true
I20250109 20:53:00.184564 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f60b81fe14f9450f8d5b0885834f3dd7" } modify_external_catalogs: true
I20250109 20:53:00.115736 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4160bb4ba40347bda48f8fa88f8b6001 (table default.Table_13f472ba51244613ab6c1c0cc5e9f9e4 [id=3fc24473de2845628a019824b932a4e6]) successfully deleted
W20250109 20:53:00.185950 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 70226.6us
I20250109 20:53:00.186659 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]) successfully deleted
I20250109 20:53:00.187484 1158 tablet_bootstrap.cc:492] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:52:59.974539 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea477dc845614ea78e71e45f862b57ff" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:52:59.977118 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 81db8c605d8944c89d9184b81de8e5d6 with delete_type TABLET_DATA_DELETED (replaced by tablet cfa50adc9f7449bb8b5eb3243b5f9405) from {username='slave'} at 127.0.0.1:57642
I20250109 20:52:59.980154 2241 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:52:59.982199 888 tablet_service.cc:1467] Processing CreateTablet for tablet 52b555c0449e4ed280c9116b151b3b7a (DEFAULT_TABLE table=default.Table_c720f45ee7754dc3ac46aaa4cbf96871 [id=1977e16013a542e087d46f0bc6abf83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:52:59.986400 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:00.188934 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b555c0449e4ed280c9116b151b3b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.191332 1158 ts_tablet_manager.cc:1397] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20250109 20:53:00.191859 1158 raft_consensus.cc:357] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.192020 1158 raft_consensus.cc:383] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.192070 1158 raft_consensus.cc:738] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.193400 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a43961d6eaec494385b2ecf67d39818b
I20250109 20:53:00.195091 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6640cef9559a4ba18d06c0aed445852f" } modify_external_catalogs: true
I20250109 20:53:00.195245 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6640cef9559a4ba18d06c0aed445852f" } modify_external_catalogs: true
I20250109 20:53:00.195752 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet eac75279378b4e6b8dc9256d67bd6dd4
I20250109 20:53:00.197346 882 tablet_service.cc:1514] Processing DeleteTablet for tablet a43961d6eaec494385b2ecf67d39818b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.197654 894 tablet_service.cc:1514] Processing DeleteTablet for tablet eac75279378b4e6b8dc9256d67bd6dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.203358 1158 consensus_queue.cc:260] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.203987 1158 ts_tablet_manager.cc:1428] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250109 20:53:00.204210 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8ed134863bf642e79c26e5bf75b248b0
I20250109 20:53:00.205039 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed134863bf642e79c26e5bf75b248b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.210909 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]). Expected version 2 got 0
I20250109 20:53:00.213146 750 tablet_service.cc:1467] Processing CreateTablet for tablet 52b555c0449e4ed280c9116b151b3b7a (DEFAULT_TABLE table=default.Table_c720f45ee7754dc3ac46aaa4cbf96871 [id=1977e16013a542e087d46f0bc6abf83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.213429 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 006475691733470f8a0468365972dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.121536 1166 ts_tablet_manager.cc:1918] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.216620 1166 log.cc:1198] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ea477dc845614ea78e71e45f862b57ff
I20250109 20:53:00.216928 1166 ts_tablet_manager.cc:1939] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:00.217446 1166 tablet_replica.cc:331] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:00.217569 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]) successfully deleted
I20250109 20:53:00.217614 1166 raft_consensus.cc:2238] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.217690 1176 tablet_bootstrap.cc:492] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:00.217730 1166 raft_consensus.cc:2267] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.217801 1176 ts_tablet_manager.cc:1397] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.275s user 0.022s sys 0.024s
W20250109 20:53:00.218103 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0019a01afbcc452a9a9374c92dc131f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0019a01afbcc452a9a9374c92dc131f3 already in progress: creating tablet
I20250109 20:53:00.218155 1166 ts_tablet_manager.cc:1905] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.218544 1176 raft_consensus.cc:357] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:00.218690 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet bba303032679462c8b16f706096b6d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bba303032679462c8b16f706096b6d4d already in progress: creating tablet
I20250109 20:53:00.218710 1176 raft_consensus.cc:383] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.218752 1176 raft_consensus.cc:738] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.218935 1176 consensus_queue.cc:260] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:00.219324 440 catalog_manager.cc:4839] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2975f30fccea4d25b3001a0b5a91d8d4 because the tablet was not found. No further retry: Not found: Tablet not found: 2975f30fccea4d25b3001a0b5a91d8d4
I20250109 20:53:00.220422 1176 ts_tablet_manager.cc:1428] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:53:00.220643 1176 tablet_bootstrap.cc:492] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:53:00.222678 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 006475691733470f8a0468365972dfd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 006475691733470f8a0468365972dfd2 already in progress: creating tablet
I20250109 20:53:00.123049 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 20d27588609f465c8dedae173e57e9b2 (DEFAULT_TABLE table=default.Table_d7b0c3e0c004488b982f368831954313 [id=2c8afc281d7c46cab5824b64446e9403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.223342 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d27588609f465c8dedae173e57e9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.123364 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d522f15b547342f4aa72cb25eb581efd (DEFAULT_TABLE table=default.Table_9ff1145eaa7f496b80139a83927b023a [id=f67740c1c16d487ab21c8bd9160dfea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.224236 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d522f15b547342f4aa72cb25eb581efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.144752 1033 tablet_service.cc:1467] Processing CreateTablet for tablet de57e94d00d748dc8b02d60f2f1fb27a (DEFAULT_TABLE table=default.Table_dc20220cc040480385a78533a6fe34e8 [id=6a133a1babbe4a4093b02bbecb1f7e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.224917 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 46df992009994d9e879109ba14bb06c0 with delete_type TABLET_DATA_DELETED (replaced by tablet d11e17f3391740b79ffa9d8a8c9bf400) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.224997 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de57e94d00d748dc8b02d60f2f1fb27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.225443 747 tablet_service.cc:1467] Processing CreateTablet for tablet a8ad256c6bf64a699ede8ff3cee5dac1 (DEFAULT_TABLE table=default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.225630 742 tablet_service.cc:1467] Processing CreateTablet for tablet 36a3d5e666f3423db8a662d99e78a99f (DEFAULT_TABLE table=default.Table_9f92e5ee4b5b4f7fa2ec542cc8ace5f7 [id=4ecefa1b1a6f4f74ba660bd519c2a142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.148999 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 7357b4f67dc6448dbb8f77dce81731f3 (DEFAULT_TABLE table=default.Table_5b016d79eedf43bbbcdc6d742dde478e [id=e030b9f57454438984429bebb8aff4a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.225817 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 2975f30fccea4d25b3001a0b5a91d8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.226020 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7357b4f67dc6448dbb8f77dce81731f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.226549 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 81db8c605d8944c89d9184b81de8e5d6 with delete_type TABLET_DATA_DELETED (replaced by tablet cfa50adc9f7449bb8b5eb3243b5f9405) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.227288 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af5cc3af3424aa3bd127450d55d6b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.227995 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 08209ea49c894209abca6c0881f534f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.228744 756 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596614521a4a92a71b781b162f70b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.158440 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
W20250109 20:53:00.190389 989 leader_election.cc:343] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.229275 989 leader_election.cc:304] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:00.229455 989 raft_consensus.cc:2681] T ea477dc845614ea78e71e45f862b57ff P 5fcc647ed479491b88e5ffacf2927a1a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:53:00.229491 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.229893 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1c9f392439440bba88a86dbd6d853e with delete_type TABLET_DATA_DELETED (replaced by tablet 9c6a06112f354465912054990ba0ea58) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.230973 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 714ef0fa6b0d4ba9b03383baec160cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.232961 756 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5e0597d9854419a16a25d1d402174c (DEFAULT_TABLE table=default.Table_6c649042616d4c449929c794c0ba36cb [id=7c8d25a44cbe47c1a98acc4a92116495]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.232946 888 tablet_service.cc:1467] Processing CreateTablet for tablet a410e84b795a4e66a554e632c56f3b43 (DEFAULT_TABLE table=default.Table_29736471ff5740c6909a38f7df420f22 [id=ae1604dcfb9f4c7bacee8ecb5bc15349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.232674 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2ca3accb8f0349d8bdd2a2306a9b1423" } modify_external_catalogs: true
I20250109 20:53:00.233302 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2ca3accb8f0349d8bdd2a2306a9b1423" } modify_external_catalogs: true
I20250109 20:53:00.234386 755 tablet_service.cc:1467] Processing CreateTablet for tablet f532f82a5696489f9240be105e03a6d5 (DEFAULT_TABLE table=default.Table_8c8845f0032e48d787d1e31838cc5c17 [id=5edb67b8f5c74744a932909df0de2be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.234582 894 tablet_service.cc:1467] Processing CreateTablet for tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2 (DEFAULT_TABLE table=default.Table_9849e2509cde44b7ad7db247abe62a79 [id=3064576ec4fb44338fa6ed3200e18ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.235905 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2975f30fccea4d25b3001a0b5a91d8d4 (DEFAULT_TABLE table=default.Table_9bebcb16a96c49088a5ac609ee22c5fb [id=ec57be40c6694016b8caa2b49931dc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.215471 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 52b555c0449e4ed280c9116b151b3b7a (DEFAULT_TABLE table=default.Table_c720f45ee7754dc3ac46aaa4cbf96871 [id=1977e16013a542e087d46f0bc6abf83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.236461 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 006475691733470f8a0468365972dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.238375 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet e75b39bbccf447ac970b9bb7a0c90a88
I20250109 20:53:00.240669 745 tablet_service.cc:1467] Processing CreateTablet for tablet eac75279378b4e6b8dc9256d67bd6dd4 (DEFAULT_TABLE table=default.Table_f60b81fe14f9450f8d5b0885834f3dd7 [id=1553a9cdf4ba4b41a348f8d1253053e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.242297 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 46df992009994d9e879109ba14bb06c0 with delete_type TABLET_DATA_DELETED (replaced by tablet d11e17f3391740b79ffa9d8a8c9bf400) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.242399 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 2975f30fccea4d25b3001a0b5a91d8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.242555 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 81db8c605d8944c89d9184b81de8e5d6 with delete_type TABLET_DATA_DELETED (replaced by tablet cfa50adc9f7449bb8b5eb3243b5f9405) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.242612 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af5cc3af3424aa3bd127450d55d6b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.242827 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1c9f392439440bba88a86dbd6d853e with delete_type TABLET_DATA_DELETED (replaced by tablet 9c6a06112f354465912054990ba0ea58) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.242980 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 714ef0fa6b0d4ba9b03383baec160cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.243233 1017 tablet_service.cc:1467] Processing CreateTablet for tablet a8ad256c6bf64a699ede8ff3cee5dac1 (DEFAULT_TABLE table=default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.243505 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ad256c6bf64a699ede8ff3cee5dac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.243839 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5e0597d9854419a16a25d1d402174c (DEFAULT_TABLE table=default.Table_6c649042616d4c449929c794c0ba36cb [id=7c8d25a44cbe47c1a98acc4a92116495]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.244184 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5e0597d9854419a16a25d1d402174c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.245100 1026 tablet_service.cc:1467] Processing CreateTablet for tablet eac75279378b4e6b8dc9256d67bd6dd4 (DEFAULT_TABLE table=default.Table_f60b81fe14f9450f8d5b0885834f3dd7 [id=1553a9cdf4ba4b41a348f8d1253053e2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:00.245214 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 54837.1us
I20250109 20:53:00.245380 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eac75279378b4e6b8dc9256d67bd6dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.245427 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 2975f30fccea4d25b3001a0b5a91d8d4 (DEFAULT_TABLE table=default.Table_9bebcb16a96c49088a5ac609ee22c5fb [id=ec57be40c6694016b8caa2b49931dc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.245807 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2975f30fccea4d25b3001a0b5a91d8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.246201 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b555c0449e4ed280c9116b151b3b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.246310 2250 raft_consensus.cc:491] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:00.246423 2250 raft_consensus.cc:513] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.247222 2250 leader_election.cc:290] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:00.242830 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 08209ea49c894209abca6c0881f534f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.247993 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8539cefebb844b3098a308ca580b0e92" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:00.248554 990 leader_election.cc:343] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.250451 1034 tablet_service.cc:1467] Processing CreateTablet for tablet f532f82a5696489f9240be105e03a6d5 (DEFAULT_TABLE table=default.Table_8c8845f0032e48d787d1e31838cc5c17 [id=5edb67b8f5c74744a932909df0de2be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.250720 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f532f82a5696489f9240be105e03a6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.254192 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_002027b48ee14395a6b054a17005d2ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.254691 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_002027b48ee14395a6b054a17005d2ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.255848 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8d34d92047e245fe92fe56a0e07f3b95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.256300 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d34d92047e245fe92fe56a0e07f3b95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.257246 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7da583c325ba439aa802225a7342b40b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.257630 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7da583c325ba439aa802225a7342b40b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.258529 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_334eff3522a142c4b245128efa66f4c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.258893 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_334eff3522a142c4b245128efa66f4c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.260676 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_41133eb367304eae9025565e3a81ce2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.261101 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41133eb367304eae9025565e3a81ce2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.262028 1158 tablet_bootstrap.cc:492] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.262332 2129 raft_consensus.cc:491] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:00.254196 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c542e77265f247738f6d277b304ffcbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:00.263541 1158 tablet_bootstrap.cc:654] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:00.265666 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c542e77265f247738f6d277b304ffcbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.266024 448 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:53:00.270414 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8539cefebb844b3098a308ca580b0e92" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:00.271893 989 leader_election.cc:343] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.272001 989 leader_election.cc:304] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.273144 2254 raft_consensus.cc:491] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:00.273262 2254 raft_consensus.cc:513] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.273785 2254 leader_election.cc:290] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:00.274247 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9b677cb92a4c4926a136962b0dac08fa" } new_table_name: "default.Table_55c4ea00225a460da552faa612cfea20" modify_external_catalogs: true
I20250109 20:53:00.274464 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9e7489c87b374499a6d3a5dded414b53" } new_table_name: "default.Table_f511041c32bb44c090802e09423715e8" modify_external_catalogs: true
I20250109 20:53:00.275372 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_80c4e31485334c5787b69339099d015c" } new_table_name: "default.Table_7598437489fd4408a5c7fafb1a722d66" modify_external_catalogs: true
I20250109 20:53:00.275486 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52b555c0449e4ed280c9116b151b3b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.276127 1157 tablet_replica.cc:331] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.276283 1157 raft_consensus.cc:2238] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.276402 1157 raft_consensus.cc:2267] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.277372 1157 ts_tablet_manager.cc:1905] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.278092 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_aa3f8eb194bf4a798c9efffc02e16f48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:00.278301 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_62c8e60a5c084b149750c3b8acb15aa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.278537 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa3f8eb194bf4a798c9efffc02e16f48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:00.278707 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62c8e60a5c084b149750c3b8acb15aa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.279668 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_486f2f72d50e49cdb7e8a1c022cb439d" } modify_external_catalogs: true
I20250109 20:53:00.279825 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_486f2f72d50e49cdb7e8a1c022cb439d" } modify_external_catalogs: true
W20250109 20:53:00.279949 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 46df992009994d9e879109ba14bb06c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46df992009994d9e879109ba14bb06c0 already in progress: creating tablet
I20250109 20:53:00.281698 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ad256c6bf64a699ede8ff3cee5dac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.281724 1165 ts_tablet_manager.cc:1428] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.181s user 0.002s sys 0.000s
I20250109 20:53:00.283064 1157 ts_tablet_manager.cc:1918] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.283039 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e9b31b27949c2948cb64b6124ee72" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:00.283147 1157 log.cc:1198] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/cd596614521a4a92a71b781b162f70b6
I20250109 20:53:00.283367 1157 ts_tablet_manager.cc:1939] T cd596614521a4a92a71b781b162f70b6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:53:00.283429 712 leader_election.cc:343] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.284322 1157 tablet_replica.cc:331] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.284478 1157 raft_consensus.cc:2238] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.284580 1157 raft_consensus.cc:2267] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.285209 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5e0597d9854419a16a25d1d402174c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.285405 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eac75279378b4e6b8dc9256d67bd6dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.287362 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a3d5e666f3423db8a662d99e78a99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.288877 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2975f30fccea4d25b3001a0b5a91d8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.292858 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f532f82a5696489f9240be105e03a6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.262430 2129 raft_consensus.cc:513] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.297484 2129 leader_election.cc:290] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:00.297624 2129 raft_consensus.cc:2743] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:00.297703 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
I20250109 20:53:00.297986 1157 ts_tablet_manager.cc:1905] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.298333 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724d15e05a864852a118badae13a5d3d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:00.298810 990 leader_election.cc:343] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.299710 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 82209a6b39bb48e791ff63a8167c9765 (table default.Table_3f42577ce23e4ed6b0f6ef3d41a6f5d0 [id=cced112727b0498f955953f2b2d29551]). Expected version 1 got 0
I20250109 20:53:00.306121 1984 raft_consensus.cc:491] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:00.306243 1984 raft_consensus.cc:513] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.306587 1157 ts_tablet_manager.cc:1918] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.306663 1157 log.cc:1198] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/0b0450d88e3e4ccaae4e5157ce5c80c1
I20250109 20:53:00.306766 1984 leader_election.cc:290] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:00.306926 1157 ts_tablet_manager.cc:1939] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:00.307518 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c9f7274a7542e89a19bd4afcf89205" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:00.307996 1158 tablet_bootstrap.cc:492] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:00.308112 1158 ts_tablet_manager.cc:1397] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.002s
W20250109 20:53:00.308378 712 leader_election.cc:343] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.308843 1158 raft_consensus.cc:357] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.309006 1158 raft_consensus.cc:383] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.309059 1158 raft_consensus.cc:738] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.309088 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e93d193814ee46d68c0d48c550964f48" } new_table_name: "default.Table_efdc83e87d8b455696e8c0aaffb7ed87" modify_external_catalogs: true
I20250109 20:53:00.309881 1158 consensus_queue.cc:260] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.310834 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:00.314049 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec1eaa6bc5a84b69840c540319265efb has no consensus state
W20250109 20:53:00.314637 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dfc973162d464694a4d59423182584ce has no consensus state
W20250109 20:53:00.315361 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bf72b6386524a4d8a6a883269abc791 consensus state has no leader
I20250109 20:53:00.319460 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a410e84b795a4e66a554e632c56f3b43. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:00.321828 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 2769e5906c604bd5b14df79db765ec61 (table default.Table_2a846b026306494b971c14609d9e438e [id=3c9451140e48432f8591841d4dfd8751], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2769e5906c604bd5b14df79db765ec61 consensus state has no leader
W20250109 20:53:00.324801 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9af5cc3af3424aa3bd127450d55d6b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af5cc3af3424aa3bd127450d55d6b36 already in progress: creating tablet
W20250109 20:53:00.327378 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 08209ea49c894209abca6c0881f534f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08209ea49c894209abca6c0881f534f5 already in progress: creating tablet
I20250109 20:53:00.327781 1160 tablet_replica.cc:331] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:00.328003 1160 raft_consensus.cc:2238] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.328187 1160 raft_consensus.cc:2267] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.329905 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f672cbd6dffc49dbb69992ae7db979ea (table default.Table_d6c99ca7f5cc4a9b8f8a47462a435ac4 [id=66c37d69f6bb4fe382c332f5c04e7a82]) successfully deleted
W20250109 20:53:00.330085 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0019a01afbcc452a9a9374c92dc131f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0019a01afbcc452a9a9374c92dc131f3 already in progress: creating tablet
W20250109 20:53:00.330272 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet bba303032679462c8b16f706096b6d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bba303032679462c8b16f706096b6d4d already in progress: creating tablet
I20250109 20:53:00.246423 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 36a3d5e666f3423db8a662d99e78a99f (DEFAULT_TABLE table=default.Table_9f92e5ee4b5b4f7fa2ec542cc8ace5f7 [id=4ecefa1b1a6f4f74ba660bd519c2a142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.330585 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 219a4b508050490cb3f1ee418e0edbf8 (table default.Table_8d504623c1894cb8843ead0ee687ea13 [id=5c7e66cff6b14f9a81029cea36224460]) successfully deleted
I20250109 20:53:00.331051 1160 ts_tablet_manager.cc:1905] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:00.331660 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 006475691733470f8a0468365972dfd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 006475691733470f8a0468365972dfd2 already in progress: creating tablet
W20250109 20:53:00.331815 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 46df992009994d9e879109ba14bb06c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46df992009994d9e879109ba14bb06c0 already in progress: creating tablet
W20250109 20:53:00.331961 441 catalog_manager.cc:4839] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2975f30fccea4d25b3001a0b5a91d8d4 because the tablet was not found. No further retry: Not found: Tablet not found: 2975f30fccea4d25b3001a0b5a91d8d4
W20250109 20:53:00.332093 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 81db8c605d8944c89d9184b81de8e5d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81db8c605d8944c89d9184b81de8e5d6 already in progress: creating tablet
W20250109 20:53:00.332229 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9af5cc3af3424aa3bd127450d55d6b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af5cc3af3424aa3bd127450d55d6b36 already in progress: creating tablet
W20250109 20:53:00.332353 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 08209ea49c894209abca6c0881f534f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08209ea49c894209abca6c0881f534f5 already in progress: creating tablet
I20250109 20:53:00.333314 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8500d6248e544e9eb829b4144818b346 (DEFAULT_TABLE table=default.Table_7d48fea31cc5403aaca9d51a191aa6d9 [id=6a2174ca2f364cfbb5157e533285beb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.333801 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8500d6248e544e9eb829b4144818b346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.338172 894 tablet_service.cc:1467] Processing CreateTablet for tablet 09ba79b42a104782bbb2443c9ffdfb2c (DEFAULT_TABLE table=default.Table_019702682d2a460fab654ee8e4fc3463 [id=2d8354f5b5ec4517bd42f8b3063cd868]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.338783 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09ba79b42a104782bbb2443c9ffdfb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.340180 1165 tablet_bootstrap.cc:492] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.341364 1165 tablet_bootstrap.cc:654] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.344427 888 tablet_service.cc:1467] Processing CreateTablet for tablet 70f06752b26c47f49d8486adc6c4d92c (DEFAULT_TABLE table=default.Table_f2388e84f29e48a9b66925c5271adcdb [id=8603063555ab4f5eaf395a5b0de69013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.344909 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f06752b26c47f49d8486adc6c4d92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.345194 1158 ts_tablet_manager.cc:1428] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20250109 20:53:00.347486 1158 tablet_bootstrap.cc:492] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.348930 1158 tablet_bootstrap.cc:654] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.354185 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a3d5e666f3423db8a662d99e78a99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.355906 1165 tablet_bootstrap.cc:492] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.356184 1165 ts_tablet_manager.cc:1397] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250109 20:53:00.360309 1166 ts_tablet_manager.cc:1918] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.360450 1166 log.cc:1198] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/bba303032679462c8b16f706096b6d4d
I20250109 20:53:00.360795 1166 ts_tablet_manager.cc:1939] T bba303032679462c8b16f706096b6d4d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:53:00.363494 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 91608.8us
I20250109 20:53:00.364113 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724d15e05a864852a118badae13a5d3d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:00.366219 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e9b31b27949c2948cb64b6124ee72" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:00.366405 1055 raft_consensus.cc:2463] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:00.366778 1176 tablet_bootstrap.cc:654] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:00.366748 713 leader_election.cc:343] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 16c9f7274a7542e89a19bd4afcf89205
I20250109 20:53:00.366876 713 leader_election.cc:304] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:00.367197 1984 raft_consensus.cc:2743] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:00.367566 713 leader_election.cc:304] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.367863 1984 raft_consensus.cc:2798] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:00.367928 1984 raft_consensus.cc:491] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:00.367982 1984 raft_consensus.cc:3054] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:00.367913 1165 raft_consensus.cc:357] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.368546 1165 raft_consensus.cc:383] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.370543 1984 raft_consensus.cc:513] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.371284 1984 leader_election.cc:290] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:00.371599 1166 tablet_replica.cc:331] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:00.368842 1165 raft_consensus.cc:738] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.371824 1166 raft_consensus.cc:2238] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.372013 1166 raft_consensus.cc:2267] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.371893 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e9b31b27949c2948cb64b6124ee72" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:00.372634 712 leader_election.cc:343] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:00.372634 989 leader_election.cc:343] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.373298 1165 consensus_queue.cc:260] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.374248 1166 ts_tablet_manager.cc:1905] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.375344 1984 raft_consensus.cc:491] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:00.375454 1984 raft_consensus.cc:513] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.376016 1984 leader_election.cc:290] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:00.372738 989 leader_election.cc:304] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.377876 2280 raft_consensus.cc:2743] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:00.378528 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e9b31b27949c2948cb64b6124ee72" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:00.378692 1051 raft_consensus.cc:3054] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:00.382122 1051 raft_consensus.cc:2463] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:00.376560 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0538f4d3608a454b9dd21a0fa42068a0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:00.383040 712 leader_election.cc:343] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.383419 713 leader_election.cc:304] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.383631 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0538f4d3608a454b9dd21a0fa42068a0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:00.383764 1051 raft_consensus.cc:2463] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:00.385299 1984 raft_consensus.cc:2798] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:00.385504 1984 raft_consensus.cc:695] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:00.385728 1984 consensus_queue.cc:237] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.387295 713 leader_election.cc:304] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.387627 2253 raft_consensus.cc:2798] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:00.387687 2253 raft_consensus.cc:491] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:00.387742 2253 raft_consensus.cc:3054] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:00.389333 1160 ts_tablet_manager.cc:1918] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.390130 2253 raft_consensus.cc:513] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.390623 2253 leader_election.cc:290] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:00.391338 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0538f4d3608a454b9dd21a0fa42068a0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:00.391839 712 leader_election.cc:343] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.391106 1160 log.cc:1198] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/a58d439254f8418091e3f23c8c274240
I20250109 20:53:00.393570 2253 raft_consensus.cc:491] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:00.393677 2253 raft_consensus.cc:513] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.394090 2253 leader_election.cc:290] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:00.393980 1165 ts_tablet_manager.cc:1428] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20250109 20:53:00.394654 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0538f4d3608a454b9dd21a0fa42068a0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:00.394824 1051 raft_consensus.cc:3054] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:00.396849 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c5a6b9681d4ef6a36e308ab2916423" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:00.397254 712 leader_election.cc:343] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:00.397827 713 leader_election.cc:343] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: f1c5a6b9681d4ef6a36e308ab2916423
I20250109 20:53:00.397915 713 leader_election.cc:304] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:00.398252 2253 raft_consensus.cc:2743] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:00.399340 1160 ts_tablet_manager.cc:1939] T a58d439254f8418091e3f23c8c274240 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:00.401551 1165 tablet_bootstrap.cc:492] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.332695 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet cd596614521a4a92a71b781b162f70b6 (table default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
I20250109 20:53:00.405247 1051 raft_consensus.cc:2463] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:00.335266 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_50d70e8a583c47059eafc658d73feb69 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 2 got 0
W20250109 20:53:00.406198 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 73499.8us
I20250109 20:53:00.367211 451 master_service.cc:946] ReplaceTablet: received request to replace tablet bdce283727d1491ba0968a17c1c00d87 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:00.406922 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 (table default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]) successfully deleted
W20250109 20:53:00.407146 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9d1c9f392439440bba88a86dbd6d853e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d1c9f392439440bba88a86dbd6d853e already in progress: creating tablet
I20250109 20:53:00.377061 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_19300a35d595402fbb4d39933c3ba00c" } new_table_name: "default.Table_eb95a1968b29430abec83ec3e35fd94b" modify_external_catalogs: true
I20250109 20:53:00.377481 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4eca95f945a544c289d09435d20f3202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.407357 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 714ef0fa6b0d4ba9b03383baec160cc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 714ef0fa6b0d4ba9b03383baec160cc8 already in progress: creating tablet
W20250109 20:53:00.407795 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4eca95f945a544c289d09435d20f3202 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.377851 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_591218a0ea174cecb8e1b74b3ee303f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:00.408068 713 leader_election.cc:304] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:00.408382 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_591218a0ea174cecb8e1b74b3ee303f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.404739 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet a58d439254f8418091e3f23c8c274240 (table default.Table_4c9330ec3e904978bb8b1f0869a0c532 [id=fbf8fa58eaee4147b61be651b770935b]) successfully deleted
I20250109 20:53:00.408445 2253 raft_consensus.cc:2798] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:00.408598 2253 raft_consensus.cc:695] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:00.408896 2253 consensus_queue.cc:237] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.410095 2280 raft_consensus.cc:491] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:00.410230 2280 raft_consensus.cc:513] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.411129 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65489a602141239c99e8e8dae22ab4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:00.411557 990 leader_election.cc:343] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.411821 2280 leader_election.cc:290] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:00.411823 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_01d2dd63e9a4480b961c408dc7d9a0a9" } modify_external_catalogs: true
I20250109 20:53:00.411455 1160 tablet_replica.cc:331] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:00.413702 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65489a602141239c99e8e8dae22ab4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:00.414212 989 leader_election.cc:343] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.414311 989 leader_election.cc:304] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.414533 2280 raft_consensus.cc:2743] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:00.415838 745 tablet_service.cc:1514] Processing DeleteTablet for tablet a43961d6eaec494385b2ecf67d39818b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.418977 1160 raft_consensus.cc:2238] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.423888 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2d8d4a28c8d342e7be15b81a4a9785ae" } modify_external_catalogs: true
I20250109 20:53:00.424057 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2d8d4a28c8d342e7be15b81a4a9785ae" } modify_external_catalogs: true
I20250109 20:53:00.425225 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d36e80f0671f404bbd0e2dded93716a3
I20250109 20:53:00.425374 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0f59949f33e14c1d8be3879d0043e1b9" } modify_external_catalogs: true
I20250109 20:53:00.425575 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0f59949f33e14c1d8be3879d0043e1b9" } modify_external_catalogs: true
I20250109 20:53:00.426261 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d36e80f0671f404bbd0e2dded93716a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:00.427136 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c70ddcc69b2b42fcbe133978dd3ac05b consensus state has no leader
I20250109 20:53:00.427162 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_01d2dd63e9a4480b961c408dc7d9a0a9" } modify_external_catalogs: true
I20250109 20:53:00.427562 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:53:00.428267 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ad256c6bf64a699ede8ff3cee5dac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.429445 745 tablet_service.cc:1514] Processing DeleteTablet for tablet eac75279378b4e6b8dc9256d67bd6dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.430370 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet eac75279378b4e6b8dc9256d67bd6dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.431245 2240 tablet.cc:1721] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:00.432648 1158 tablet_bootstrap.cc:492] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:00.432783 1158 ts_tablet_manager.cc:1397] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.000s
I20250109 20:53:00.421460 1160 raft_consensus.cc:2267] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:00.433153 2032 raft_consensus.cc:668] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20250109 20:53:00.432649 894 tablet_service.cc:1514] Processing DeleteTablet for tablet b405d16cc62444f2ad94fb69bb9fb6ee with delete_type TABLET_DATA_DELETED (replaced by tablet d9f18f244a344272bba2bc085686b61d) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.433480 1158 raft_consensus.cc:357] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.433645 1158 raft_consensus.cc:383] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.433692 1158 raft_consensus.cc:738] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.433879 1158 consensus_queue.cc:260] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.434396 1158 ts_tablet_manager.cc:1428] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:00.434597 1158 tablet_bootstrap.cc:492] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.436131 1158 tablet_bootstrap.cc:654] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.436297 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet a43961d6eaec494385b2ecf67d39818b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.437399 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed134863bf642e79c26e5bf75b248b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.437640 1157 tablet_replica.cc:331] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.437813 1157 raft_consensus.cc:2238] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.437932 1157 raft_consensus.cc:2267] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.436385 2242 tablet.cc:1721] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:00.439765 1157 ts_tablet_manager.cc:1905] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.440016 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0504dfd389e44bafa7837757c1cec711" } modify_external_catalogs: true
I20250109 20:53:00.440167 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0504dfd389e44bafa7837757c1cec711" } modify_external_catalogs: true
I20250109 20:53:00.442096 745 tablet_service.cc:1514] Processing DeleteTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.442443 745 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.442759 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.443080 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 with delete_type TABLET_DATA_DELETED (replaced by tablet 56fac585b2904ab490a6882dd5252a9a) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.443519 745 tablet_service.cc:1467] Processing CreateTablet for tablet 02b5127f0b7848038e48c879a9785618 (DEFAULT_TABLE table=default.Table_5533e631d0dc4529b038c22187a08afd [id=bdd87101f968403499714582c97b09b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.443823 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b5127f0b7848038e48c879a9785618. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:00.444087 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f933f9582c24c63b9893da59e5a66d0 consensus state has no leader
I20250109 20:53:00.444599 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8190cb332af34a398ae8970567562d0e
I20250109 20:53:00.445266 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba
I20250109 20:53:00.446252 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.449909 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0e4d5613811444c38ea786a6ad43f9c1
I20250109 20:53:00.450670 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4d5613811444c38ea786a6ad43f9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.434121 1176 tablet_bootstrap.cc:492] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:00.455346 1176 ts_tablet_manager.cc:1397] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.235s user 0.006s sys 0.001s
I20250109 20:53:00.456112 1176 raft_consensus.cc:357] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.456570 1157 ts_tablet_manager.cc:1918] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.456703 1157 log.cc:1198] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/8ed134863bf642e79c26e5bf75b248b0
I20250109 20:53:00.457141 1157 ts_tablet_manager.cc:1939] T 8ed134863bf642e79c26e5bf75b248b0 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:00.460152 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8190cb332af34a398ae8970567562d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.461174 1157 tablet_replica.cc:331] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.461452 1157 raft_consensus.cc:2238] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.461606 1157 raft_consensus.cc:2267] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.461908 1160 ts_tablet_manager.cc:1905] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.462810 1157 ts_tablet_manager.cc:1905] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.437191 1166 ts_tablet_manager.cc:1918] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.462983 1166 log.cc:1198] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/85a95a31468b431fb4304cf6680016da
I20250109 20:53:00.463519 1166 ts_tablet_manager.cc:1939] T 85a95a31468b431fb4304cf6680016da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:00.464157 1165 tablet_bootstrap.cc:654] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.465485 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 02b5127f0b7848038e48c879a9785618 (DEFAULT_TABLE table=default.Table_5533e631d0dc4529b038c22187a08afd [id=bdd87101f968403499714582c97b09b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.465909 1984 consensus_queue.cc:1035] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:00.471179 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e85779bce1cf4ab4926e6c7af5b7fa33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.471709 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e85779bce1cf4ab4926e6c7af5b7fa33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.475396 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:53:00.475633 451 catalog_manager.cc:5526] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:00.476087 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:00.476294 451 catalog_manager.cc:5526] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:00.476604 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]). Expected version 3 got 0
I20250109 20:53:00.477550 751 tablet_service.cc:1467] Processing CreateTablet for tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2 (DEFAULT_TABLE table=default.Table_9849e2509cde44b7ad7db247abe62a79 [id=3064576ec4fb44338fa6ed3200e18ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.477509 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_50f7e7cb701c4134ae4844b388bf458a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:00.477880 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:00.477928 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50f7e7cb701c4134ae4844b388bf458a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.478868 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6c494755f29b4b4ab961151e05604fa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.479276 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c494755f29b4b4ab961151e05604fa9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.485172 745 tablet_service.cc:1467] Processing CreateTablet for tablet a410e84b795a4e66a554e632c56f3b43 (DEFAULT_TABLE table=default.Table_29736471ff5740c6909a38f7df420f22 [id=ae1604dcfb9f4c7bacee8ecb5bc15349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.485572 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a410e84b795a4e66a554e632c56f3b43. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:00.492905 712 consensus_peers.cc:487] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:00.437413 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed134863bf642e79c26e5bf75b248b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.476150 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a410e84b795a4e66a554e632c56f3b43 (DEFAULT_TABLE table=default.Table_29736471ff5740c6909a38f7df420f22 [id=ae1604dcfb9f4c7bacee8ecb5bc15349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.476958 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2 (DEFAULT_TABLE table=default.Table_9849e2509cde44b7ad7db247abe62a79 [id=3064576ec4fb44338fa6ed3200e18ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.496766 455 master_service.cc:946] ReplaceTablet: received request to replace tablet a1ee36f3be324b14868976298ca26118 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:00.497572 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 8500d6248e544e9eb829b4144818b346 (DEFAULT_TABLE table=default.Table_7d48fea31cc5403aaca9d51a191aa6d9 [id=6a2174ca2f364cfbb5157e533285beb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.498313 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 09ba79b42a104782bbb2443c9ffdfb2c (DEFAULT_TABLE table=default.Table_019702682d2a460fab654ee8e4fc3463 [id=2d8354f5b5ec4517bd42f8b3063cd868]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.498670 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ea477dc845614ea78e71e45f862b57ff (table default.Table_1b9aa886edd542fc8039fd921d1048c8 [id=d64f7249495e433fa9c57277ee5eaba9]) successfully deleted
I20250109 20:53:00.498682 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 70f06752b26c47f49d8486adc6c4d92c (DEFAULT_TABLE table=default.Table_f2388e84f29e48a9b66925c5271adcdb [id=8603063555ab4f5eaf395a5b0de69013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.500402 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet bba303032679462c8b16f706096b6d4d (table default.Table_eb7bd4795ee740f0ba9088dc52e7b383 [id=6a25ad9a50db4cd7b62f73b4e268fdae]) successfully deleted
I20250109 20:53:00.501201 751 tablet_service.cc:1467] Processing CreateTablet for tablet 8500d6248e544e9eb829b4144818b346 (DEFAULT_TABLE table=default.Table_7d48fea31cc5403aaca9d51a191aa6d9 [id=6a2174ca2f364cfbb5157e533285beb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.501408 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 881ea20c17904d669c8a1fbd766ba559
I20250109 20:53:00.501631 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8500d6248e544e9eb829b4144818b346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.505385 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d8dc3eb6434742bd9f61b5989fa3a6c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.505918 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8dc3eb6434742bd9f61b5989fa3a6c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.510254 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a1ee36f3be324b14868976298ca26118
I20250109 20:53:00.510350 755 tablet_service.cc:1467] Processing CreateTablet for tablet 70f06752b26c47f49d8486adc6c4d92c (DEFAULT_TABLE table=default.Table_f2388e84f29e48a9b66925c5271adcdb [id=8603063555ab4f5eaf395a5b0de69013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.510658 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f06752b26c47f49d8486adc6c4d92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.510668 455 catalog_manager.cc:6596] ReplaceTablet: tablet a1ee36f3be324b14868976298ca26118 deleted and replaced by tablet 407c57dfd4134eeba07f74386540f844
I20250109 20:53:00.512727 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_50d70e8a583c47059eafc658d73feb69 [id=ee4103ad12894523a9f260bc6ff91419] alter complete (version 2)
I20250109 20:53:00.481992 1166 tablet_replica.cc:331] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:00.517064 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2a846b026306494b971c14609d9e438e" } new_table_name: "default.Table_f59d59762b8b4f55b3123c4df5bfd12e" modify_external_catalogs: true
I20250109 20:53:00.517666 1166 raft_consensus.cc:2238] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.517843 1166 raft_consensus.cc:2267] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.518328 1166 ts_tablet_manager.cc:1905] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.521591 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_684bc1fac0764a80acc3c9be2f5df7ea" } new_table_name: "default.Table_210d37e8a6044352be29c7f936e49338" modify_external_catalogs: true
I20250109 20:53:00.523145 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_97cc80276bc548db859d3adb65d9bfa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.523749 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97cc80276bc548db859d3adb65d9bfa9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.525442 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f45cf4921b02449da770ba575ad3043d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:00.523038 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a410e84b795a4e66a554e632c56f3b43. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:00.525996 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f45cf4921b02449da770ba575ad3043d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.527070 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c4498a12cf0e4b99b80384de1e0a7bd2" } new_table_name: "default.Table_e05a32da191c46ffb04a2fa03c7920f5" modify_external_catalogs: true
I20250109 20:53:00.528851 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.529764 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bc2f052f8a5c4572b2f58622e6dc5f29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.530185 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc2f052f8a5c4572b2f58622e6dc5f29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.533143 894 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ee36f3be324b14868976298ca26118 with delete_type TABLET_DATA_DELETED (replaced by tablet 407c57dfd4134eeba07f74386540f844) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:00.536103 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2769e5906c604bd5b14df79db765ec61 consensus state has no leader
I20250109 20:53:00.536355 751 tablet_service.cc:1467] Processing CreateTablet for tablet 09ba79b42a104782bbb2443c9ffdfb2c (DEFAULT_TABLE table=default.Table_019702682d2a460fab654ee8e4fc3463 [id=2d8354f5b5ec4517bd42f8b3063cd868]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.536568 1158 tablet_bootstrap.cc:492] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:00.536823 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09ba79b42a104782bbb2443c9ffdfb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.536973 1158 ts_tablet_manager.cc:1397] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.011s
I20250109 20:53:00.537643 1158 raft_consensus.cc:357] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.537806 1158 raft_consensus.cc:383] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.537843 1158 raft_consensus.cc:738] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.534965 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 881ea20c17904d669c8a1fbd766ba559 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.538034 1158 consensus_queue.cc:260] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:00.539898 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632 consensus state has no leader
I20250109 20:53:00.539984 1158 ts_tablet_manager.cc:1428] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:53:00.540210 1158 tablet_bootstrap.cc:492] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:00.540416 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 646a6db049704d4e8169af36b15edbe0 consensus state has no leader
I20250109 20:53:00.540953 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b5127f0b7848038e48c879a9785618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.541177 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f06752b26c47f49d8486adc6c4d92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.541886 1158 tablet_bootstrap.cc:654] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.544972 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09ba79b42a104782bbb2443c9ffdfb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.549865 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8500d6248e544e9eb829b4144818b346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.551115 1157 ts_tablet_manager.cc:1918] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.551205 1157 log.cc:1198] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f1c5a6b9681d4ef6a36e308ab2916423
I20250109 20:53:00.551443 1157 ts_tablet_manager.cc:1939] T f1c5a6b9681d4ef6a36e308ab2916423 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:00.552167 1157 tablet_replica.cc:331] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.552325 1157 raft_consensus.cc:2238] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.552429 1157 raft_consensus.cc:2267] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.552835 1160 ts_tablet_manager.cc:1918] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.553419 1160 log.cc:1198] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4025cdef634d4626a33f01c362a1388a
I20250109 20:53:00.555236 1160 ts_tablet_manager.cc:1939] T 4025cdef634d4626a33f01c362a1388a P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:00.557233 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4025cdef634d4626a33f01c362a1388a (table default.Table_d52320f2397e4526ae456263f71bcda2 [id=0b96e2cb7e234f498b8ef5e9a09e9a67]) successfully deleted
W20250109 20:53:00.558678 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 714ef0fa6b0d4ba9b03383baec160cc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 714ef0fa6b0d4ba9b03383baec160cc8 already in progress: creating tablet
W20250109 20:53:00.560046 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9d1c9f392439440bba88a86dbd6d853e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d1c9f392439440bba88a86dbd6d853e already in progress: creating tablet
W20250109 20:53:00.563434 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 81db8c605d8944c89d9184b81de8e5d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81db8c605d8944c89d9184b81de8e5d6 already in progress: creating tablet
W20250109 20:53:00.564630 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a43961d6eaec494385b2ecf67d39818b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a43961d6eaec494385b2ecf67d39818b already in progress: creating tablet
I20250109 20:53:00.566020 1984 raft_consensus.cc:491] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:53:00.566126 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet eac75279378b4e6b8dc9256d67bd6dd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eac75279378b4e6b8dc9256d67bd6dd4 already in progress: creating tablet
I20250109 20:53:00.566165 1984 raft_consensus.cc:513] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.566727 1984 leader_election.cc:290] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:00.567222 1166 ts_tablet_manager.cc:1918] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:53:00.567348 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8ed134863bf642e79c26e5bf75b248b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ed134863bf642e79c26e5bf75b248b0 already in progress: creating tablet
I20250109 20:53:00.567463 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda300a7e58a44339c3bdb2e62a9f384" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:00.567890 712 leader_election.cc:343] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:00.568856 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d36e80f0671f404bbd0e2dded93716a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36e80f0671f404bbd0e2dded93716a3 already in progress: creating tablet
I20250109 20:53:00.569594 1160 tablet_replica.cc:331] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:00.570608 1160 raft_consensus.cc:2238] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.571409 1160 raft_consensus.cc:2267] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.573872 1160 ts_tablet_manager.cc:1905] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:00.575868 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0019a01afbcc452a9a9374c92dc131f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0019a01afbcc452a9a9374c92dc131f3 already in progress: creating tablet
I20250109 20:53:00.576201 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 189cd8079a9d4ca6b1703f4361d5ec5d from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:00.576340 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 85a95a31468b431fb4304cf6680016da (table default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]) successfully deleted
W20250109 20:53:00.579885 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a43961d6eaec494385b2ecf67d39818b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a43961d6eaec494385b2ecf67d39818b already in progress: creating tablet
W20250109 20:53:00.580098 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet eac75279378b4e6b8dc9256d67bd6dd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eac75279378b4e6b8dc9256d67bd6dd4 already in progress: creating tablet
I20250109 20:53:00.580411 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 8ed134863bf642e79c26e5bf75b248b0 (table default.Table_6640cef9559a4ba18d06c0aed445852f [id=46c011650d964e26b40be8a854e1f8c0]) successfully deleted
I20250109 20:53:00.568867 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda300a7e58a44339c3bdb2e62a9f384" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:00.580868 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f1c5a6b9681d4ef6a36e308ab2916423 (table default.Table_68ffdf1aeefe420ea24aa1102012f77c [id=4d826d307bff4ab2b43a9d8b718055d6]) successfully deleted
I20250109 20:53:00.580899 1051 raft_consensus.cc:2463] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:00.577366 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet d36e80f0671f404bbd0e2dded93716a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.577947 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8190cb332af34a398ae8970567562d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.581182 713 leader_election.cc:304] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.578328 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.581415 1166 log.cc:1198] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/225046d45c1c43b488babf5da8de75d7
I20250109 20:53:00.581637 1984 raft_consensus.cc:2798] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:00.581696 1166 ts_tablet_manager.cc:1939] T 225046d45c1c43b488babf5da8de75d7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:00.581743 1984 raft_consensus.cc:491] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:00.581857 1984 raft_consensus.cc:3054] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:00.582257 1157 ts_tablet_manager.cc:1905] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.579092 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 881ea20c17904d669c8a1fbd766ba559 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.579437 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ee36f3be324b14868976298ca26118 with delete_type TABLET_DATA_DELETED (replaced by tablet 407c57dfd4134eeba07f74386540f844) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.588860 1165 tablet_bootstrap.cc:492] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.588976 1165 ts_tablet_manager.cc:1397] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.189s user 0.013s sys 0.001s
I20250109 20:53:00.589618 1165 raft_consensus.cc:357] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.589815 1165 raft_consensus.cc:383] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.589874 1165 raft_consensus.cc:738] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.590085 1165 consensus_queue.cc:260] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.590505 1165 ts_tablet_manager.cc:1428] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 20:53:00.578708 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4d5613811444c38ea786a6ad43f9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.590982 1165 tablet_bootstrap.cc:492] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.582537 755 tablet_service.cc:1514] Processing DeleteTablet for tablet d36e80f0671f404bbd0e2dded93716a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.582788 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8190cb332af34a398ae8970567562d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.582886 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.582930 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4d5613811444c38ea786a6ad43f9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.583132 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 881ea20c17904d669c8a1fbd766ba559 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.583352 754 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ee36f3be324b14868976298ca26118 with delete_type TABLET_DATA_DELETED (replaced by tablet 407c57dfd4134eeba07f74386540f844) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.583474 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.583493 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a39fac67894a208dd7ec3153b05e4a with delete_type TABLET_DATA_DELETED (replaced by tablet 9629fe30752f49478afad9b93dc235eb) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.584060 1984 raft_consensus.cc:513] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.593060 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda300a7e58a44339c3bdb2e62a9f384" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:00.593521 712 leader_election.cc:343] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.593804 1984 leader_election.cc:290] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:00.595729 1165 tablet_bootstrap.cc:654] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.597272 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 705a7c04f3f948e1a5e486d5ccd48ccc with delete_type TABLET_DATA_DELETED (replaced by tablet 829bec4f66c24180afd5271057a7d0e8) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.599279 1166 tablet_replica.cc:331] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:00.599623 1166 raft_consensus.cc:2238] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.600016 1166 raft_consensus.cc:2267] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.600808 1166 ts_tablet_manager.cc:1905] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.601464 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 01174f43d56c4c96a3b754828fc30640 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:00.601015 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda300a7e58a44339c3bdb2e62a9f384" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:00.601766 1051 raft_consensus.cc:3054] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:00.603338 1160 ts_tablet_manager.cc:1918] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.603740 1160 log.cc:1198] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/b405d16cc62444f2ad94fb69bb9fb6ee
I20250109 20:53:00.604270 1051 raft_consensus.cc:2463] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:00.604598 713 leader_election.cc:304] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.604789 1160 ts_tablet_manager.cc:1939] T b405d16cc62444f2ad94fb69bb9fb6ee P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:00.605121 1984 raft_consensus.cc:2798] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:00.605326 1984 raft_consensus.cc:695] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:00.605880 1984 consensus_queue.cc:237] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.606410 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet b405d16cc62444f2ad94fb69bb9fb6ee (table default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]) successfully deleted
I20250109 20:53:00.607079 1160 tablet_replica.cc:331] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:00.607698 1160 raft_consensus.cc:2238] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.608256 1160 raft_consensus.cc:2267] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.612601 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 225046d45c1c43b488babf5da8de75d7 (table default.Table_711789d44eb242309002efa5bd06ffb1 [id=a638ba7e8a3b4c309df4fd05ca948518]) successfully deleted
I20250109 20:53:00.614509 1158 tablet_bootstrap.cc:492] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:00.614665 1158 ts_tablet_manager.cc:1397] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.006s
I20250109 20:53:00.615511 1158 raft_consensus.cc:357] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.615757 1158 raft_consensus.cc:383] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.615953 1158 raft_consensus.cc:738] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.616219 1158 consensus_queue.cc:260] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.618610 1158 ts_tablet_manager.cc:1428] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:00.619390 1158 tablet_bootstrap.cc:492] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.621349 1158 tablet_bootstrap.cc:654] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.624507 454 catalog_manager.cc:5526] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:00.625798 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:53:00.632580 1176 raft_consensus.cc:383] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.632957 1157 ts_tablet_manager.cc:1918] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.633116 1157 log.cc:1198] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/e0f67c4294f8499496db4e82db8c3bf0
I20250109 20:53:00.633422 1157 ts_tablet_manager.cc:1939] T e0f67c4294f8499496db4e82db8c3bf0 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:00.634338 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet e0f67c4294f8499496db4e82db8c3bf0 (table default.Table_bf29c4a64f7b4d49af314ad6211982b1 [id=7d02be4e89a7426ea835a9041c394fd4]) successfully deleted
I20250109 20:53:00.634554 1176 raft_consensus.cc:738] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.634886 1176 consensus_queue.cc:260] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.635107 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cb6e0dd9d8ff4cde977360e8da20fd10" } new_table_name: "default.Table_7f4fb38109404eeaa5a743e859c813c6" modify_external_catalogs: true
I20250109 20:53:00.636662 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cf33c8d126254dd580ef3fb97ecad07f" } modify_external_catalogs: true
I20250109 20:53:00.636871 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cf33c8d126254dd580ef3fb97ecad07f" } modify_external_catalogs: true
I20250109 20:53:00.642016 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet f03d4e679a1c4793bd83cd979acced88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.643996 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9558dfe563d94482bf496e19b8a80edc with delete_type TABLET_DATA_DELETED (replaced by tablet a8ad256c6bf64a699ede8ff3cee5dac1) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.644929 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a4b4139a715c423ea5d28aa9c087d962"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.645618 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4b4139a715c423ea5d28aa9c087d962 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.646507 1176 ts_tablet_manager.cc:1428] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.191s user 0.016s sys 0.000s
I20250109 20:53:00.646737 1157 tablet_replica.cc:331] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.646970 1157 raft_consensus.cc:2238] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.647210 1157 raft_consensus.cc:2267] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.647181 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_625afa33e7894342beb6fceaae41f1b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.649046 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_625afa33e7894342beb6fceaae41f1b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.650512 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_770bfdf999cd4d70b649d37f9f16197e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.650918 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_770bfdf999cd4d70b649d37f9f16197e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.652153 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_65bd13ddff1b46929045b98b3c549a18" } modify_external_catalogs: true
I20250109 20:53:00.652328 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_65bd13ddff1b46929045b98b3c549a18" } modify_external_catalogs: true
I20250109 20:53:00.652936 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3" } modify_external_catalogs: true
I20250109 20:53:00.653075 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3" } modify_external_catalogs: true
I20250109 20:53:00.654042 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_eb95a1968b29430abec83ec3e35fd94b" } new_table_name: "default.Table_77ae678d30f14a639c449b8d3735f66f" modify_external_catalogs: true
I20250109 20:53:00.657641 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a1280d01e103484e88adcd24c412c519" } modify_external_catalogs: true
I20250109 20:53:00.657757 1176 tablet_bootstrap.cc:492] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:00.657976 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a1280d01e103484e88adcd24c412c519" } modify_external_catalogs: true
I20250109 20:53:00.635869 1166 ts_tablet_manager.cc:1918] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.658321 1166 log.cc:1198] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/8539cefebb844b3098a308ca580b0e92
I20250109 20:53:00.659469 1166 ts_tablet_manager.cc:1939] T 8539cefebb844b3098a308ca580b0e92 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:00.660571 1157 ts_tablet_manager.cc:1905] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.661521 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 8539cefebb844b3098a308ca580b0e92 (table default.Table_1d52e3d34dc14e729a515dc24e0787d4 [id=b74c21f2cb874f2485c0e3bc1d6e0dc8]) successfully deleted
W20250109 20:53:00.662611 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8c0f5dc3c794811b55040dcc094246c consensus state has no leader
I20250109 20:53:00.662854 1166 tablet_replica.cc:331] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:00.663177 1166 raft_consensus.cc:2238] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.663336 1166 raft_consensus.cc:2267] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.665872 1051 raft_consensus.cc:1270] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:53:00.666517 2253 consensus_queue.cc:1035] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:00.666630 1166 ts_tablet_manager.cc:1905] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:00.667317 712 consensus_peers.cc:487] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:00.672731 1158 tablet_bootstrap.cc:492] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
W20250109 20:53:00.672884 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f933f9582c24c63b9893da59e5a66d0 consensus state has no leader
I20250109 20:53:00.672895 1158 ts_tablet_manager.cc:1397] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.016s
I20250109 20:53:00.673704 1158 raft_consensus.cc:357] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.673866 1158 raft_consensus.cc:383] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.673923 1158 raft_consensus.cc:738] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.673902 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b6bcf34f4788483783acb3d09accba37
I20250109 20:53:00.674271 1158 consensus_queue.cc:260] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.674499 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bcf34f4788483783acb3d09accba37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.675532 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bcf34f4788483783acb3d09accba37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:00.678694 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bcf34f4788483783acb3d09accba37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.679289 1160 ts_tablet_manager.cc:1905] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.679517 1165 tablet_bootstrap.cc:492] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.679612 1165 ts_tablet_manager.cc:1397] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.002s
I20250109 20:53:00.680239 1165 raft_consensus.cc:357] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.680397 1165 raft_consensus.cc:383] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.680439 1165 raft_consensus.cc:738] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.680604 1165 consensus_queue.cc:260] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.681054 1165 ts_tablet_manager.cc:1428] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:53:00.681303 1165 tablet_bootstrap.cc:492] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.682608 1165 tablet_bootstrap.cc:654] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:00.683585 2223 log.cc:926] Time spent T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250109 20:53:00.685091 1158 ts_tablet_manager.cc:1428] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250109 20:53:00.686893 1158 tablet_bootstrap.cc:492] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.692729 1158 tablet_bootstrap.cc:654] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.697026 2280 raft_consensus.cc:491] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:00.697177 2280 raft_consensus.cc:513] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.697881 2280 leader_election.cc:290] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:00.698464 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82209a6b39bb48e791ff63a8167c9765" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:00.698848 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82209a6b39bb48e791ff63a8167c9765" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:00.699504 989 leader_election.cc:343] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:00.699784 990 leader_election.cc:343] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:00.699882 990 leader_election.cc:304] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:00.699947 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 with delete_type TABLET_DATA_DELETED (replaced by tablet b6b23623677b4e088c14af925b4e07d4) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.700328 2280 raft_consensus.cc:2743] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:00.703423 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_532e9f42a7084d3c9037828c563a3f7b" } modify_external_catalogs: true
I20250109 20:53:00.703605 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_532e9f42a7084d3c9037828c563a3f7b" } modify_external_catalogs: true
I20250109 20:53:00.708456 1165 tablet_bootstrap.cc:492] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.708581 1165 ts_tablet_manager.cc:1397] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.002s
I20250109 20:53:00.709419 1165 raft_consensus.cc:357] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.709594 1165 raft_consensus.cc:383] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.709652 1165 raft_consensus.cc:738] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.709877 1165 consensus_queue.cc:260] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.710335 1165 ts_tablet_manager.cc:1428] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:00.710608 1165 tablet_bootstrap.cc:492] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.712513 1165 tablet_bootstrap.cc:654] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.713996 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 2b1891cbef8f4b0cae0c15d478db4860 (table default.Table_ca51464e265c4e28b9e54fe7be0587fc [id=0a0599668d77405ba0e72eda2aa5219e]). Expected version 4 got 0
I20250109 20:53:00.716617 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e4bfa188b545349f358adfd0bc714b with delete_type TABLET_DATA_DELETED (replaced by tablet a1ed00f7954143c9ba9bf65d8a8a68d2) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.717075 2240 tablet.cc:1721] T 2b1891cbef8f4b0cae0c15d478db4860 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:00.720036 2242 tablet.cc:1721] T 2b1891cbef8f4b0cae0c15d478db4860 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:00.720645 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0f0e855cab8543789fddd27edca161f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.721228 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f0e855cab8543789fddd27edca161f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.724503 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:00.730780 1166 ts_tablet_manager.cc:1918] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.730966 1166 log.cc:1198] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/724d15e05a864852a118badae13a5d3d
I20250109 20:53:00.731309 1166 ts_tablet_manager.cc:1939] T 724d15e05a864852a118badae13a5d3d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:00.731612 1157 ts_tablet_manager.cc:1918] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.731782 1157 log.cc:1198] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/73e82535780c47ca8b9cd94b013b4b42
I20250109 20:53:00.732116 1157 ts_tablet_manager.cc:1939] T 73e82535780c47ca8b9cd94b013b4b42 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:00.733083 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 73e82535780c47ca8b9cd94b013b4b42 (table default.Table_f7702e93eb894ea5bbf0af322f94c29c [id=0193ea72702947b590c27b63fd70f93f]) successfully deleted
I20250109 20:53:00.733390 1157 tablet_replica.cc:331] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.733613 1157 raft_consensus.cc:2238] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.733880 1157 raft_consensus.cc:2267] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.734462 1157 ts_tablet_manager.cc:1905] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.734701 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_ca51464e265c4e28b9e54fe7be0587fc [id=0a0599668d77405ba0e72eda2aa5219e] alter complete (version 4)
I20250109 20:53:00.737666 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_628c3b9194114d9d920428c59ced645e" } new_table_name: "default.Table_0ded6ea82440418fb3dbbd259fb0cc77" modify_external_catalogs: true
I20250109 20:53:00.740442 1176 tablet_bootstrap.cc:654] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:00.743321 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 643e337667d34534b88715704ab88e82 consensus state has no leader
I20250109 20:53:00.743490 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b6729d452f65409992e893d661ace4db" } new_table_name: "default.Table_57ac5b97c0f941e6bfa7c994e2d81482" modify_external_catalogs: true
I20250109 20:53:00.743330 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3dbeca6078054ff0877acae499fb525c" } new_table_name: "default.Table_a2b81fc56f9c469e9b72ed20afcb7e02" modify_external_catalogs: true
W20250109 20:53:00.749805 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b53c48fd0e0146cebf19d62519d1f773 has no consensus state
W20250109 20:53:00.750644 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b0e82dbd939465ca7044f0aa1f46f06 consensus state has no leader
I20250109 20:53:00.760613 1160 ts_tablet_manager.cc:1918] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:00.761381 1160 log.cc:1198] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/776e6e2b3b9a4f9f9d08361ba1cbb5ba
I20250109 20:53:00.763626 1160 ts_tablet_manager.cc:1939] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:00.766729 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]) successfully deleted
W20250109 20:53:00.768240 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0e4d5613811444c38ea786a6ad43f9c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e4d5613811444c38ea786a6ad43f9c1 already in progress: creating tablet
W20250109 20:53:00.769675 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8190cb332af34a398ae8970567562d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8190cb332af34a398ae8970567562d0e already in progress: creating tablet
W20250109 20:53:00.770888 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a1ee36f3be324b14868976298ca26118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ee36f3be324b14868976298ca26118 already in progress: creating tablet
I20250109 20:53:00.771785 1160 tablet_replica.cc:331] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:00.772689 1160 raft_consensus.cc:2238] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.774005 1160 raft_consensus.cc:2267] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.775797 1158 tablet_bootstrap.cc:492] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:00.776463 1158 ts_tablet_manager.cc:1397] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.011s
I20250109 20:53:00.777633 1160 ts_tablet_manager.cc:1905] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.778448 1158 raft_consensus.cc:357] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.778867 1158 raft_consensus.cc:383] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.779032 1158 raft_consensus.cc:738] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.779281 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 724d15e05a864852a118badae13a5d3d (table default.Table_5c9e541186c64e158515aff3e19ccda9 [id=9fcc13c731ac492fbf382b79a3df9ed2]) successfully deleted
I20250109 20:53:00.779454 1158 consensus_queue.cc:260] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.781040 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:00.782572 2240 tablet.cc:1721] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:00.783509 2242 tablet.cc:1721] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:00.784266 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 with delete_type TABLET_DATA_DELETED (replaced by tablet d79437d800f44c5a8222eea7dc2cd4e3) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:00.785991 1158 ts_tablet_manager.cc:1428] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250109 20:53:00.788771 1158 tablet_bootstrap.cc:492] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:00.789399 1176 tablet_bootstrap.cc:492] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:00.789539 1176 ts_tablet_manager.cc:1397] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.132s user 0.007s sys 0.002s
I20250109 20:53:00.790144 1176 raft_consensus.cc:357] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.790338 1176 raft_consensus.cc:383] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.790421 1176 raft_consensus.cc:738] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.790647 1176 consensus_queue.cc:260] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.791730 1158 tablet_bootstrap.cc:654] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.796573 1176 ts_tablet_manager.cc:1428] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250109 20:53:00.807765 1165 tablet_bootstrap.cc:492] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.807905 1165 ts_tablet_manager.cc:1397] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.007s
I20250109 20:53:00.808599 1165 raft_consensus.cc:357] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.808835 1165 raft_consensus.cc:383] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.808895 1165 raft_consensus.cc:738] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.809108 1165 consensus_queue.cc:260] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.809546 1165 ts_tablet_manager.cc:1428] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:00.809794 1165 tablet_bootstrap.cc:492] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.816825 891 tablet_service.cc:1514] Processing DeleteTablet for tablet af8c0bf27e994e6aa4cbedca16febe12 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cb88771d2b148bdae6e427c1b25b9a9) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:00.818156 1165 tablet_bootstrap.cc:654] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.819705 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3a0f3311269245aa8308597dae6b0428"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.820230 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a0f3311269245aa8308597dae6b0428 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.823390 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f] alter complete (version 2)
I20250109 20:53:00.825003 1160 ts_tablet_manager.cc:1918] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:00.825858 1160 log.cc:1198] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/881ea20c17904d669c8a1fbd766ba559
I20250109 20:53:00.826864 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4b14acbb79d24c09be2fe93f8f1e2c16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:00.827086 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8c8b587893cb4890887373eb89950b96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.827360 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b14acbb79d24c09be2fe93f8f1e2c16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:00.827498 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c8b587893cb4890887373eb89950b96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.828558 1176 tablet_bootstrap.cc:492] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:00.828888 1160 ts_tablet_manager.cc:1939] T 881ea20c17904d669c8a1fbd766ba559 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:00.830183 1176 tablet_bootstrap.cc:654] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.831490 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d9871f7468704e5982b38ab253e14fef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:00.832163 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_0504dfd389e44bafa7837757c1cec711 [id=3e073742d9cf476387b0a0949059d933]) successfully deleted
I20250109 20:53:00.832602 1160 tablet_replica.cc:331] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
W20250109 20:53:00.833524 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 006475691733470f8a0468365972dfd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 006475691733470f8a0468365972dfd2 already in progress: creating tablet
W20250109 20:53:00.833766 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9871f7468704e5982b38ab253e14fef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.835738 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_97f699f90bdf400b9f6c3ea5616fff83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:00.839740 1160 raft_consensus.cc:2238] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.838387 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_99b31f42d9d14193b7b8b9809d39c458"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.840590 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99b31f42d9d14193b7b8b9809d39c458 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.841151 1160 raft_consensus.cc:2267] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:00.842356 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97f699f90bdf400b9f6c3ea5616fff83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:00.845059 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 46df992009994d9e879109ba14bb06c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46df992009994d9e879109ba14bb06c0 already in progress: creating tablet
W20250109 20:53:00.845712 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2975f30fccea4d25b3001a0b5a91d8d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2975f30fccea4d25b3001a0b5a91d8d4 already in progress: creating tablet
W20250109 20:53:00.846297 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 81db8c605d8944c89d9184b81de8e5d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81db8c605d8944c89d9184b81de8e5d6 already in progress: creating tablet
I20250109 20:53:00.846251 1160 ts_tablet_manager.cc:1905] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:00.846879 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9af5cc3af3424aa3bd127450d55d6b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af5cc3af3424aa3bd127450d55d6b36 already in progress: creating tablet
W20250109 20:53:00.847368 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9d1c9f392439440bba88a86dbd6d853e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d1c9f392439440bba88a86dbd6d853e already in progress: creating tablet
W20250109 20:53:00.847852 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 714ef0fa6b0d4ba9b03383baec160cc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 714ef0fa6b0d4ba9b03383baec160cc8 already in progress: creating tablet
W20250109 20:53:00.848367 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 08209ea49c894209abca6c0881f534f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08209ea49c894209abca6c0881f534f5 already in progress: creating tablet
W20250109 20:53:00.849095 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet eac75279378b4e6b8dc9256d67bd6dd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eac75279378b4e6b8dc9256d67bd6dd4 already in progress: creating tablet
I20250109 20:53:00.850276 1166 tablet_replica.cc:331] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:00.850623 1166 raft_consensus.cc:2238] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.850929 1166 raft_consensus.cc:2267] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.851730 1166 ts_tablet_manager.cc:1905] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.853135 915 raft_consensus.cc:3054] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:00.849982 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a43961d6eaec494385b2ecf67d39818b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a43961d6eaec494385b2ecf67d39818b already in progress: creating tablet
I20250109 20:53:00.863787 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_613b9c78c2024fb58046e80d8ad29df8" } new_table_name: "default.Table_59ec3dfe97bc4ffab283a5868c5d01dc" modify_external_catalogs: true
W20250109 20:53:00.870683 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 086c6b621cae4214bcec4daa1a725272 consensus state has no leader
I20250109 20:53:00.871052 1160 ts_tablet_manager.cc:1918] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.871161 1160 log.cc:1198] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/705a7c04f3f948e1a5e486d5ccd48ccc
I20250109 20:53:00.871445 1160 ts_tablet_manager.cc:1939] T 705a7c04f3f948e1a5e486d5ccd48ccc P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:00.871968 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 705a7c04f3f948e1a5e486d5ccd48ccc (table default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]) successfully deleted
I20250109 20:53:00.872082 1160 tablet_replica.cc:331] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:00.872239 1160 raft_consensus.cc:2238] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:00.872351 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b6bcf34f4788483783acb3d09accba37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6bcf34f4788483783acb3d09accba37 already in progress: creating tablet
I20250109 20:53:00.872375 1160 raft_consensus.cc:2267] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.873510 446 master_service.cc:946] ReplaceTablet: received request to replace tablet d40b8ce042f349d599ce2a3bd2e4a91d from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:00.874152 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_20316b8f5dec42f191b3384673593c52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.874557 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20316b8f5dec42f191b3384673593c52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.875555 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_392b71f0c7604df7acc42d6729b4d78a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:00.875963 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_392b71f0c7604df7acc42d6729b4d78a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:00.877321 1176 tablet_bootstrap.cc:492] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:00.877503 1176 ts_tablet_manager.cc:1397] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20250109 20:53:00.878225 1176 raft_consensus.cc:357] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.878511 1176 raft_consensus.cc:383] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.878610 1176 raft_consensus.cc:738] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.878902 1176 consensus_queue.cc:260] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.879544 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_50d70e8a583c47059eafc658d73feb69 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 2 got 0
I20250109 20:53:00.882014 1176 ts_tablet_manager.cc:1428] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:00.882361 1176 tablet_bootstrap.cc:492] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:00.882704 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_50d70e8a583c47059eafc658d73feb69 [id=ee4103ad12894523a9f260bc6ff91419]). Expected version 2 got 0
I20250109 20:53:00.886368 2308 tablet.cc:1721] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:00.890759 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:00.893484 1165 tablet_bootstrap.cc:492] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.893628 1165 ts_tablet_manager.cc:1397] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.009s
I20250109 20:53:00.894462 1165 raft_consensus.cc:357] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.894652 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (table default.Table_159068abc4d842c693e5ae2323d1147d [id=15c9f7d5d3ff4fa4865b55c5892421fb]). Expected version 3 got 0
I20250109 20:53:00.894739 1165 raft_consensus.cc:383] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.894802 1165 raft_consensus.cc:738] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.867235 1157 ts_tablet_manager.cc:1918] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.894987 1157 log.cc:1198] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/16c9f7274a7542e89a19bd4afcf89205
I20250109 20:53:00.895066 1165 consensus_queue.cc:260] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.895572 1157 ts_tablet_manager.cc:1939] T 16c9f7274a7542e89a19bd4afcf89205 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:00.898901 891 tablet_service.cc:1467] Processing CreateTablet for tablet caa4c29aff0f4914b8495c83d952c3a6 (DEFAULT_TABLE table=default.Table_c542e77265f247738f6d277b304ffcbd [id=3803b4552d904b99a9bb1e57d2ba700e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.899236 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa4c29aff0f4914b8495c83d952c3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.900697 1157 tablet_replica.cc:331] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:00.900992 1157 raft_consensus.cc:2238] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:00.901346 1157 raft_consensus.cc:2267] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.901633 1160 ts_tablet_manager.cc:1905] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.901997 1157 ts_tablet_manager.cc:1905] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.902657 1165 ts_tablet_manager.cc:1428] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250109 20:53:00.906211 888 tablet_service.cc:1467] Processing CreateTablet for tablet cfb05218aecf461aa0b9dfbd769adf84 (DEFAULT_TABLE table=default.Table_41133eb367304eae9025565e3a81ce2c [id=01109f22dc9543f3830f20a4f7634101]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.906523 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb05218aecf461aa0b9dfbd769adf84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.908946 894 tablet_service.cc:1467] Processing CreateTablet for tablet 0e392ffa3384491da72da4b713e4a132 (DEFAULT_TABLE table=default.Table_334eff3522a142c4b245128efa66f4c6 [id=9b5e3f5c73d442e0a2f2660a99edf556]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.909245 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e392ffa3384491da72da4b713e4a132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.911232 1165 tablet_bootstrap.cc:492] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.912971 885 tablet_service.cc:1467] Processing CreateTablet for tablet 4597aa225a374487b13647961b1843d8 (DEFAULT_TABLE table=default.Table_7da583c325ba439aa802225a7342b40b [id=784d7914837e4b6888ff9b7da4bdc858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.913308 1165 tablet_bootstrap.cc:654] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.913472 883 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae81d22bb904903a75590beca4b7ee2 (DEFAULT_TABLE table=default.Table_8d34d92047e245fe92fe56a0e07f3b95 [id=4d978f6dbe98424b89f46b02f93aec02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.913841 884 tablet_service.cc:1467] Processing CreateTablet for tablet d0dc4addb7fd4d23822b25af5e288118 (DEFAULT_TABLE table=default.Table_002027b48ee14395a6b054a17005d2ab [id=8845af37fb3c4d2e8a523cdf50cd872d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.914173 890 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce2173203d4dc4910065f830b335ef (DEFAULT_TABLE table=default.Table_026d505e4cfa48c69084663bbc553f83 [id=4e354ce426174bc1bafd4d694a59d20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.914872 1166 ts_tablet_manager.cc:1918] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.915006 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4597aa225a374487b13647961b1843d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.915064 1166 log.cc:1198] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/8ed134863bf642e79c26e5bf75b248b0
I20250109 20:53:00.915406 1166 ts_tablet_manager.cc:1939] T 8ed134863bf642e79c26e5bf75b248b0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:00.921228 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ce2173203d4dc4910065f830b335ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.921780 893 tablet_service.cc:1467] Processing CreateTablet for tablet ec1eaa6bc5a84b69840c540319265efb (DEFAULT_TABLE table=default.Table_f511041c32bb44c090802e09423715e8 [id=2b6f40a6c2ef4ab9af48b1671e0e45fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.922439 1176 tablet_bootstrap.cc:654] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.924324 881 tablet_service.cc:1467] Processing CreateTablet for tablet ab7d4f31211742e3b23e9f22e36cf7c7 (DEFAULT_TABLE table=default.Table_86fde8a5cdef40daa1e0e95b33e93710 [id=e4ceb81d0a0c41d9b1ff82f50b6bcb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.926213 1160 ts_tablet_manager.cc:1918] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:00.926265 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1eaa6bc5a84b69840c540319265efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.926311 1160 log.cc:1198] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/66e4bfa188b545349f358adfd0bc714b
I20250109 20:53:00.926592 1160 ts_tablet_manager.cc:1939] T 66e4bfa188b545349f358adfd0bc714b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:00.927022 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 66e4bfa188b545349f358adfd0bc714b (table default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]) successfully deleted
I20250109 20:53:00.928131 887 tablet_service.cc:1467] Processing CreateTablet for tablet b53c48fd0e0146cebf19d62519d1f773 (DEFAULT_TABLE table=default.Table_57ac5b97c0f941e6bfa7c994e2d81482 [id=3b2dbbd43d9d45359f1a081f5f61214e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.928467 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b53c48fd0e0146cebf19d62519d1f773. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.929333 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae81d22bb904903a75590beca4b7ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.929463 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0dc4addb7fd4d23822b25af5e288118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.932804 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7d4f31211742e3b23e9f22e36cf7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.932971 882 tablet_service.cc:1467] Processing CreateTablet for tablet e19d64024eb94df69b2c5d52791bb4c6 (DEFAULT_TABLE table=default.Table_c75db1b93cbd4edb9fce068615c63752 [id=8bb5d0a21e5046e9b384c4a50a109264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.933306 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19d64024eb94df69b2c5d52791bb4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.933526 1158 tablet_bootstrap.cc:492] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:00.933658 1158 ts_tablet_manager.cc:1397] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.145s user 0.002s sys 0.002s
I20250109 20:53:00.934566 1158 raft_consensus.cc:357] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.934749 1158 raft_consensus.cc:383] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.934814 1158 raft_consensus.cc:738] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.936096 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 8ed134863bf642e79c26e5bf75b248b0 (table default.Table_6640cef9559a4ba18d06c0aed445852f [id=46c011650d964e26b40be8a854e1f8c0]) successfully deleted
I20250109 20:53:00.936283 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 16c9f7274a7542e89a19bd4afcf89205 (table default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]) successfully deleted
I20250109 20:53:00.936357 1020 tablet_service.cc:1467] Processing CreateTablet for tablet caa4c29aff0f4914b8495c83d952c3a6 (DEFAULT_TABLE table=default.Table_c542e77265f247738f6d277b304ffcbd [id=3803b4552d904b99a9bb1e57d2ba700e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:00.936429 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d36e80f0671f404bbd0e2dded93716a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36e80f0671f404bbd0e2dded93716a3 already in progress: creating tablet
I20250109 20:53:00.936704 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa4c29aff0f4914b8495c83d952c3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.937471 751 tablet_service.cc:1467] Processing CreateTablet for tablet caa4c29aff0f4914b8495c83d952c3a6 (DEFAULT_TABLE table=default.Table_c542e77265f247738f6d277b304ffcbd [id=3803b4552d904b99a9bb1e57d2ba700e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.937650 756 tablet_service.cc:1467] Processing CreateTablet for tablet 0e392ffa3384491da72da4b713e4a132 (DEFAULT_TABLE table=default.Table_334eff3522a142c4b245128efa66f4c6 [id=9b5e3f5c73d442e0a2f2660a99edf556]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.937796 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa4c29aff0f4914b8495c83d952c3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.938071 740 tablet_service.cc:1467] Processing CreateTablet for tablet 4597aa225a374487b13647961b1843d8 (DEFAULT_TABLE table=default.Table_7da583c325ba439aa802225a7342b40b [id=784d7914837e4b6888ff9b7da4bdc858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.938351 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4597aa225a374487b13647961b1843d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.940600 745 tablet_service.cc:1467] Processing CreateTablet for tablet cfb05218aecf461aa0b9dfbd769adf84 (DEFAULT_TABLE table=default.Table_41133eb367304eae9025565e3a81ce2c [id=01109f22dc9543f3830f20a4f7634101]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.940902 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb05218aecf461aa0b9dfbd769adf84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.940923 746 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae81d22bb904903a75590beca4b7ee2 (DEFAULT_TABLE table=default.Table_8d34d92047e245fe92fe56a0e07f3b95 [id=4d978f6dbe98424b89f46b02f93aec02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.940990 1019 tablet_service.cc:1467] Processing CreateTablet for tablet cfb05218aecf461aa0b9dfbd769adf84 (DEFAULT_TABLE table=default.Table_41133eb367304eae9025565e3a81ce2c [id=01109f22dc9543f3830f20a4f7634101]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.941190 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae81d22bb904903a75590beca4b7ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.941319 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb05218aecf461aa0b9dfbd769adf84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.944965 755 tablet_service.cc:1467] Processing CreateTablet for tablet b53c48fd0e0146cebf19d62519d1f773 (DEFAULT_TABLE table=default.Table_57ac5b97c0f941e6bfa7c994e2d81482 [id=3b2dbbd43d9d45359f1a081f5f61214e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.944989 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 0e392ffa3384491da72da4b713e4a132 (DEFAULT_TABLE table=default.Table_334eff3522a142c4b245128efa66f4c6 [id=9b5e3f5c73d442e0a2f2660a99edf556]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.945144 750 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce2173203d4dc4910065f830b335ef (DEFAULT_TABLE table=default.Table_026d505e4cfa48c69084663bbc553f83 [id=4e354ce426174bc1bafd4d694a59d20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.945259 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b53c48fd0e0146cebf19d62519d1f773. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.945291 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e392ffa3384491da72da4b713e4a132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.945398 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ce2173203d4dc4910065f830b335ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.944965 754 tablet_service.cc:1467] Processing CreateTablet for tablet d0dc4addb7fd4d23822b25af5e288118 (DEFAULT_TABLE table=default.Table_002027b48ee14395a6b054a17005d2ab [id=8845af37fb3c4d2e8a523cdf50cd872d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.947121 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0dc4addb7fd4d23822b25af5e288118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.948745 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae81d22bb904903a75590beca4b7ee2 (DEFAULT_TABLE table=default.Table_8d34d92047e245fe92fe56a0e07f3b95 [id=4d978f6dbe98424b89f46b02f93aec02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.948949 747 tablet_service.cc:1467] Processing CreateTablet for tablet ec1eaa6bc5a84b69840c540319265efb (DEFAULT_TABLE table=default.Table_f511041c32bb44c090802e09423715e8 [id=2b6f40a6c2ef4ab9af48b1671e0e45fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.949115 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d0dc4addb7fd4d23822b25af5e288118 (DEFAULT_TABLE table=default.Table_002027b48ee14395a6b054a17005d2ab [id=8845af37fb3c4d2e8a523cdf50cd872d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.949229 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1eaa6bc5a84b69840c540319265efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.949465 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce2173203d4dc4910065f830b335ef (DEFAULT_TABLE table=default.Table_026d505e4cfa48c69084663bbc553f83 [id=4e354ce426174bc1bafd4d694a59d20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.949805 1016 tablet_service.cc:1467] Processing CreateTablet for tablet ec1eaa6bc5a84b69840c540319265efb (DEFAULT_TABLE table=default.Table_f511041c32bb44c090802e09423715e8 [id=2b6f40a6c2ef4ab9af48b1671e0e45fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.950119 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ab7d4f31211742e3b23e9f22e36cf7c7 (DEFAULT_TABLE table=default.Table_86fde8a5cdef40daa1e0e95b33e93710 [id=e4ceb81d0a0c41d9b1ff82f50b6bcb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.950438 1032 tablet_service.cc:1467] Processing CreateTablet for tablet b53c48fd0e0146cebf19d62519d1f773 (DEFAULT_TABLE table=default.Table_57ac5b97c0f941e6bfa7c994e2d81482 [id=3b2dbbd43d9d45359f1a081f5f61214e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.951331 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 4597aa225a374487b13647961b1843d8 (DEFAULT_TABLE table=default.Table_7da583c325ba439aa802225a7342b40b [id=784d7914837e4b6888ff9b7da4bdc858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.952677 1165 tablet_bootstrap.cc:492] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.952847 1165 ts_tablet_manager.cc:1397] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.001s
I20250109 20:53:00.952934 742 tablet_service.cc:1467] Processing CreateTablet for tablet ab7d4f31211742e3b23e9f22e36cf7c7 (DEFAULT_TABLE table=default.Table_86fde8a5cdef40daa1e0e95b33e93710 [id=e4ceb81d0a0c41d9b1ff82f50b6bcb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.953204 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7d4f31211742e3b23e9f22e36cf7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.953583 1165 raft_consensus.cc:357] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.953768 1165 raft_consensus.cc:383] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.953817 1165 raft_consensus.cc:738] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.957376 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae81d22bb904903a75590beca4b7ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.957787 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e392ffa3384491da72da4b713e4a132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.958422 1165 consensus_queue.cc:260] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.960980 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0dc4addb7fd4d23822b25af5e288118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.964835 1165 ts_tablet_manager.cc:1428] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250109 20:53:00.964946 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ce2173203d4dc4910065f830b335ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.965142 1165 tablet_bootstrap.cc:492] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:00.966380 881 tablet_service.cc:1467] Processing CreateTablet for tablet 64d462969643467c97222cac8adc210d (DEFAULT_TABLE table=default.Table_8a70d90c9fb04010bfc9fb5342c82862 [id=b3a59ec3fdf04d5792216b2f8859ce23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.966859 1165 tablet_bootstrap.cc:654] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:00.966930 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d462969643467c97222cac8adc210d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.969023 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1eaa6bc5a84b69840c540319265efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.973031 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7d4f31211742e3b23e9f22e36cf7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.974555 881 tablet_service.cc:1467] Processing CreateTablet for tablet 04b4247f52ce4d33bf49d9625a1592bf (DEFAULT_TABLE table=default.Table_bd1103b105f94221b1bed1edfe6477e3 [id=412d889629a84f50aca3788e40f2916e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.975090 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b53c48fd0e0146cebf19d62519d1f773. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.979017 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4597aa225a374487b13647961b1843d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.980299 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b4247f52ce4d33bf49d9625a1592bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.984014 1160 tablet_replica.cc:331] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:00.984169 1160 raft_consensus.cc:2238] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:00.984288 1160 raft_consensus.cc:2267] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:00.985170 1160 ts_tablet_manager.cc:1905] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:00.986168 1165 tablet_bootstrap.cc:492] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:00.986310 1165 ts_tablet_manager.cc:1397] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250109 20:53:00.987054 1165 raft_consensus.cc:357] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.987229 1165 raft_consensus.cc:383] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:00.987278 1165 raft_consensus.cc:738] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:00.987481 1165 consensus_queue.cc:260] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:00.988525 1017 tablet_service.cc:1467] Processing CreateTablet for tablet e19d64024eb94df69b2c5d52791bb4c6 (DEFAULT_TABLE table=default.Table_c75db1b93cbd4edb9fce068615c63752 [id=8bb5d0a21e5046e9b384c4a50a109264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.988920 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19d64024eb94df69b2c5d52791bb4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.996116 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 64d462969643467c97222cac8adc210d (DEFAULT_TABLE table=default.Table_8a70d90c9fb04010bfc9fb5342c82862 [id=b3a59ec3fdf04d5792216b2f8859ce23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.996556 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d462969643467c97222cac8adc210d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.998234 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 04b4247f52ce4d33bf49d9625a1592bf (DEFAULT_TABLE table=default.Table_bd1103b105f94221b1bed1edfe6477e3 [id=412d889629a84f50aca3788e40f2916e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:00.998610 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b4247f52ce4d33bf49d9625a1592bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:00.999709 894 tablet_service.cc:1467] Processing CreateTablet for tablet cfa50adc9f7449bb8b5eb3243b5f9405 (DEFAULT_TABLE table=default.Table_ae7ad8e7e47a4b3fa7011f6db7700313 [id=116eadc3ef1c42ec85ef73873a9bf69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.000062 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa50adc9f7449bb8b5eb3243b5f9405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.000284 746 tablet_service.cc:1467] Processing CreateTablet for tablet 64d462969643467c97222cac8adc210d (DEFAULT_TABLE table=default.Table_8a70d90c9fb04010bfc9fb5342c82862 [id=b3a59ec3fdf04d5792216b2f8859ce23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.000648 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d462969643467c97222cac8adc210d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.004510 1026 tablet_service.cc:1467] Processing CreateTablet for tablet edfcbc74f489406297bfa8df228d2787 (DEFAULT_TABLE table=default.Table_75f8ca17cad34737a386cc58f15a0697 [id=0f3556c8dc804cad897c10a2b0140169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.005471 747 tablet_service.cc:1467] Processing CreateTablet for tablet e19d64024eb94df69b2c5d52791bb4c6 (DEFAULT_TABLE table=default.Table_c75db1b93cbd4edb9fce068615c63752 [id=8bb5d0a21e5046e9b384c4a50a109264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.006381 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19d64024eb94df69b2c5d52791bb4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.007411 881 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfbf64b178841d7aaf34632cbff3716 (DEFAULT_TABLE table=default.Table_2ff361cf1b484434b63f79b7ae7da8a2 [id=b52e753aee1e47af81de2ff1bcbea690]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.007961 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab297e2c76349cb97273200f6a950d8 (DEFAULT_TABLE table=default.Table_d1574f6ea52544dd8cc0cec65fd89000 [id=9fa2bd0445924df48b0a922a965b6fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.008318 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab297e2c76349cb97273200f6a950d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.008419 742 tablet_service.cc:1467] Processing CreateTablet for tablet 04b4247f52ce4d33bf49d9625a1592bf (DEFAULT_TABLE table=default.Table_bd1103b105f94221b1bed1edfe6477e3 [id=412d889629a84f50aca3788e40f2916e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.008941 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b4247f52ce4d33bf49d9625a1592bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.009366 1984 raft_consensus.cc:491] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.009483 1984 raft_consensus.cc:513] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.010066 1984 leader_election.cc:290] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:01.011114 713 leader_election.cc:343] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 35a39fac67894a208dd7ec3153b05e4a
I20250109 20:53:01.012012 756 tablet_service.cc:1467] Processing CreateTablet for tablet edfcbc74f489406297bfa8df228d2787 (DEFAULT_TABLE table=default.Table_75f8ca17cad34737a386cc58f15a0697 [id=0f3556c8dc804cad897c10a2b0140169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.012351 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfcbc74f489406297bfa8df228d2787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.012758 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfcbc74f489406297bfa8df228d2787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.017390 750 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab297e2c76349cb97273200f6a950d8 (DEFAULT_TABLE table=default.Table_d1574f6ea52544dd8cc0cec65fd89000 [id=9fa2bd0445924df48b0a922a965b6fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.017879 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab297e2c76349cb97273200f6a950d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.022951 747 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8344dbb1c540abadd22531e19a52d1 (DEFAULT_TABLE table=default.Table_5157484d0a55469184fc46834bcf66bb [id=8db0f4c51882416c9e47bd567cb5949d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.023212 1030 tablet_service.cc:1467] Processing CreateTablet for tablet cfa50adc9f7449bb8b5eb3243b5f9405 (DEFAULT_TABLE table=default.Table_ae7ad8e7e47a4b3fa7011f6db7700313 [id=116eadc3ef1c42ec85ef73873a9bf69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.023413 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8344dbb1c540abadd22531e19a52d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.023581 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa50adc9f7449bb8b5eb3243b5f9405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.024456 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8344dbb1c540abadd22531e19a52d1 (DEFAULT_TABLE table=default.Table_5157484d0a55469184fc46834bcf66bb [id=8db0f4c51882416c9e47bd567cb5949d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.024794 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8344dbb1c540abadd22531e19a52d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.027374 740 tablet_service.cc:1467] Processing CreateTablet for tablet cfa50adc9f7449bb8b5eb3243b5f9405 (DEFAULT_TABLE table=default.Table_ae7ad8e7e47a4b3fa7011f6db7700313 [id=116eadc3ef1c42ec85ef73873a9bf69a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.027730 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa50adc9f7449bb8b5eb3243b5f9405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.033558 1166 tablet_replica.cc:331] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:01.033823 1166 raft_consensus.cc:2238] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.034021 1166 raft_consensus.cc:2267] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:01.034471 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d36e80f0671f404bbd0e2dded93716a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36e80f0671f404bbd0e2dded93716a3 already in progress: creating tablet
W20250109 20:53:01.034706 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8190cb332af34a398ae8970567562d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8190cb332af34a398ae8970567562d0e already in progress: creating tablet
I20250109 20:53:01.047098 1158 consensus_queue.cc:260] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.047744 1158 ts_tablet_manager.cc:1428] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.114s user 0.001s sys 0.001s
I20250109 20:53:01.055665 1157 ts_tablet_manager.cc:1918] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:01.055819 1157 log.cc:1198] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/776e6e2b3b9a4f9f9d08361ba1cbb5ba
I20250109 20:53:01.056275 1157 ts_tablet_manager.cc:1939] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.056749 1158 tablet_bootstrap.cc:492] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:01.058591 1176 tablet_bootstrap.cc:492] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.058770 1176 ts_tablet_manager.cc:1397] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.176s user 0.001s sys 0.001s
I20250109 20:53:01.059630 1176 raft_consensus.cc:357] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.059895 1176 raft_consensus.cc:383] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.059993 1176 raft_consensus.cc:738] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.060257 1176 consensus_queue.cc:260] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.061312 2280 raft_consensus.cc:491] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.061411 2280 raft_consensus.cc:513] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.061929 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (table default.Table_159068abc4d842c693e5ae2323d1147d [id=15c9f7d5d3ff4fa4865b55c5892421fb]). Expected version 3 got 0
I20250109 20:53:01.061977 2280 leader_election.cc:290] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:01.062104 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:01.062449 1176 ts_tablet_manager.cc:1428] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:01.062393 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c57d7e4c6798461280d6ff32c28b05f7" } new_table_name: "default.Table_f90416c39947490a86198dc9ac982cca" modify_external_catalogs: true
I20250109 20:53:01.062598 1166 ts_tablet_manager.cc:1905] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.062752 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03d4e679a1c4793bd83cd979acced88" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:01.062808 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_99e81a65fa2741ac80eccc081148192a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.063244 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99e81a65fa2741ac80eccc081148192a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:01.063608 990 leader_election.cc:343] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:01.066972 448 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5ce0a941f0c14501a947a5c7e2a844e8 consensus state has no leader
I20250109 20:53:01.068176 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_58817df2014d4f20bc1db1e6d113a63e" } modify_external_catalogs: true
I20250109 20:53:01.068351 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_58817df2014d4f20bc1db1e6d113a63e" } modify_external_catalogs: true
I20250109 20:53:01.073514 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 01174f43d56c4c96a3b754828fc30640
I20250109 20:53:01.073911 451 catalog_manager.cc:6596] ReplaceTablet: tablet 01174f43d56c4c96a3b754828fc30640 deleted and replaced by tablet fdb0ebfad2de4fbf89a4b53a0a150906
I20250109 20:53:01.074041 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 01174f43d56c4c96a3b754828fc30640 with delete_type TABLET_DATA_DELETED (replaced by tablet fdb0ebfad2de4fbf89a4b53a0a150906) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.077391 1176 tablet_bootstrap.cc:492] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:01.079157 1176 tablet_bootstrap.cc:654] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.080034 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 01174f43d56c4c96a3b754828fc30640 with delete_type TABLET_DATA_DELETED (replaced by tablet fdb0ebfad2de4fbf89a4b53a0a150906) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.081230 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0ccc75429f7441f1b186879bc5d12f92
I20250109 20:53:01.081597 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 11dbbc9ceb1a4252b1605560bdb9b62a
I20250109 20:53:01.081809 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_62601ef08e864a52b863ba7c0d6230ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.082080 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1b296a6b3a0c47eb8dead9e581ac987b" } new_table_name: "default.Table_e778e2d95d0240ef965b87c9f44fc0ba" modify_external_catalogs: true
W20250109 20:53:01.082212 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62601ef08e864a52b863ba7c0d6230ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.083020 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3af9563f6efd47059caac7c86c41d43f
I20250109 20:53:01.083021 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ccc75429f7441f1b186879bc5d12f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.083196 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 11dbbc9ceb1a4252b1605560bdb9b62a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.083400 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a6006590b7764efc901bf51ca4e19ab3
I20250109 20:53:01.083513 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f3548a3fd5ff4cf0867cf1de1bb898c6" } new_table_name: "default.Table_a074d0a90e09441f884e98409d8f5e4b" modify_external_catalogs: true
I20250109 20:53:01.083827 1984 raft_consensus.cc:491] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.083865 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_47985b842e6c4ab6adaf55ef480a90d2" } new_table_name: "default.Table_4ed34f82b8c04d07b91df48899ffdfc2" modify_external_catalogs: true
I20250109 20:53:01.083927 1984 raft_consensus.cc:513] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.084138 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af9563f6efd47059caac7c86c41d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.084475 1984 leader_election.cc:290] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.084636 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a6006590b7764efc901bf51ca4e19ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.085589 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af9563f6efd47059caac7c86c41d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:01.086190 713 leader_election.cc:343] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2a4f047e8fa34ca79e78915c61a8ba12
I20250109 20:53:01.086730 747 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfbf64b178841d7aaf34632cbff3716 (DEFAULT_TABLE table=default.Table_2ff361cf1b484434b63f79b7ae7da8a2 [id=b52e753aee1e47af81de2ff1bcbea690]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.086642 451 master_service.cc:946] ReplaceTablet: received request to replace tablet fb3e620799a943c6850e20a3d6d2f6bb from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:01.087637 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet a6006590b7764efc901bf51ca4e19ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.087867 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 11dbbc9ceb1a4252b1605560bdb9b62a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.086841 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d40b8ce042f349d599ce2a3bd2e4a91d
I20250109 20:53:01.088042 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ccc75429f7441f1b186879bc5d12f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.088021 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.088372 446 catalog_manager.cc:6596] ReplaceTablet: tablet d40b8ce042f349d599ce2a3bd2e4a91d deleted and replaced by tablet 59ec9a54da2e43bfbb22637385c53a7a
I20250109 20:53:01.088518 1158 tablet_bootstrap.cc:654] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.088624 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b1b59379d39a4c639dcc4611e0f60767"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.089097 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1b59379d39a4c639dcc4611e0f60767 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.090579 740 tablet_service.cc:1514] Processing DeleteTablet for tablet d40b8ce042f349d599ce2a3bd2e4a91d with delete_type TABLET_DATA_DELETED (replaced by tablet 59ec9a54da2e43bfbb22637385c53a7a) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.090993 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfbf64b178841d7aaf34632cbff3716 (DEFAULT_TABLE table=default.Table_2ff361cf1b484434b63f79b7ae7da8a2 [id=b52e753aee1e47af81de2ff1bcbea690]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.091596 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfbf64b178841d7aaf34632cbff3716. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:01.094915 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c368e39a840a4eceaf17180255fda3e9 consensus state has no leader
I20250109 20:53:01.095402 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfbf64b178841d7aaf34632cbff3716. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.098397 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_311b8a2f631245c9979c5be620fd1791"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.098865 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_311b8a2f631245c9979c5be620fd1791 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.099284 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_908fd543e71a4471a5c0f402e7ac5a7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.099835 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet d40b8ce042f349d599ce2a3bd2e4a91d with delete_type TABLET_DATA_DELETED (replaced by tablet 59ec9a54da2e43bfbb22637385c53a7a) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:01.099954 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_908fd543e71a4471a5c0f402e7ac5a7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.102398 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9d6b86869c9849318edb0e8e4360f246"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.102813 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d6b86869c9849318edb0e8e4360f246 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.010573 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a39fac67894a208dd7ec3153b05e4a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:01.016882 888 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab297e2c76349cb97273200f6a950d8 (DEFAULT_TABLE table=default.Table_d1574f6ea52544dd8cc0cec65fd89000 [id=9fa2bd0445924df48b0a922a965b6fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.105628 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab297e2c76349cb97273200f6a950d8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:01.105639 712 leader_election.cc:343] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.105753 712 leader_election.cc:304] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.106278 2254 raft_consensus.cc:2743] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.107407 740 tablet_service.cc:1467] Processing CreateTablet for tablet 7a5c88e044c64f638a7959b6dffea612 (DEFAULT_TABLE table=default.Table_7f04a87f321a4355bb2fc4cbfb0b9076 [id=e51ccdcd9fd543be913a7b884f112387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.021164 890 tablet_service.cc:1467] Processing CreateTablet for tablet 7b8344dbb1c540abadd22531e19a52d1 (DEFAULT_TABLE table=default.Table_5157484d0a55469184fc46834bcf66bb [id=8db0f4c51882416c9e47bd567cb5949d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.111207 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8344dbb1c540abadd22531e19a52d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.115178 2280 raft_consensus.cc:491] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.115324 2280 raft_consensus.cc:513] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.115952 2280 leader_election.cc:290] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:01.116034 1166 ts_tablet_manager.cc:1918] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:01.116127 1166 log.cc:1198] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/776e6e2b3b9a4f9f9d08361ba1cbb5ba
I20250109 20:53:01.116529 1166 ts_tablet_manager.cc:1939] T 776e6e2b3b9a4f9f9d08361ba1cbb5ba P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:01.116487 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cd9cd48910497ba3e4b159583cb7a6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:01.117070 989 leader_election.cc:343] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.117515 1166 tablet_replica.cc:331] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:01.117703 1166 raft_consensus.cc:2238] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.117681 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]) successfully deleted
I20250109 20:53:01.117874 1166 raft_consensus.cc:2267] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:01.117959 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a1ee36f3be324b14868976298ca26118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ee36f3be324b14868976298ca26118 already in progress: creating tablet
I20250109 20:53:01.022029 883 tablet_service.cc:1467] Processing CreateTablet for tablet edfcbc74f489406297bfa8df228d2787 (DEFAULT_TABLE table=default.Table_75f8ca17cad34737a386cc58f15a0697 [id=0f3556c8dc804cad897c10a2b0140169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.063854 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03d4e679a1c4793bd83cd979acced88" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:01.118381 1166 ts_tablet_manager.cc:1905] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.065178 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1630e4438e4337ba026b41f630a480 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.118803 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfcbc74f489406297bfa8df228d2787. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:01.119987 989 leader_election.cc:343] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.120083 989 leader_election.cc:304] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:01.120460 2280 raft_consensus.cc:2743] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.080057 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 01174f43d56c4c96a3b754828fc30640 with delete_type TABLET_DATA_DELETED (replaced by tablet fdb0ebfad2de4fbf89a4b53a0a150906) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.085644 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af9563f6efd47059caac7c86c41d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.085675 887 tablet_service.cc:1514] Processing DeleteTablet for tablet a6006590b7764efc901bf51ca4e19ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.085850 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4f047e8fa34ca79e78915c61a8ba12" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:01.121202 712 leader_election.cc:343] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.121305 712 leader_election.cc:304] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.121666 2254 raft_consensus.cc:2743] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.123128 887 tablet_service.cc:1467] Processing CreateTablet for tablet affb5b43f6314e08b0deb3b2e8b40207 (DEFAULT_TABLE table=default.Table_42be9454af31421f84024e4849874065 [id=efed9da4e675477dacd24addf44968dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.123517 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affb5b43f6314e08b0deb3b2e8b40207. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.088356 1165 ts_tablet_manager.cc:1428] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.102s user 0.001s sys 0.001s
I20250109 20:53:01.091450 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 11dbbc9ceb1a4252b1605560bdb9b62a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.124132 1165 tablet_bootstrap.cc:492] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:01.092419 882 tablet_service.cc:1514] Processing DeleteTablet for tablet d40b8ce042f349d599ce2a3bd2e4a91d with delete_type TABLET_DATA_DELETED (replaced by tablet 59ec9a54da2e43bfbb22637385c53a7a) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.092975 895 tablet_service.cc:1467] Processing CreateTablet for tablet 7a5c88e044c64f638a7959b6dffea612 (DEFAULT_TABLE table=default.Table_7f04a87f321a4355bb2fc4cbfb0b9076 [id=e51ccdcd9fd543be913a7b884f112387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.093124 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ccc75429f7441f1b186879bc5d12f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.093684 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f215eee55d4e6a97a384df6ff9d596 with delete_type TABLET_DATA_DELETED (replaced by tablet e16a1d9a5b374b8dbf725eb32ffadb9e) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.099779 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5ea5d588834f1b8b2b057ec324aadc with delete_type TABLET_DATA_DELETED (replaced by tablet cd23012af5574f37944f9e26c07d05d2) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.125032 1158 tablet_bootstrap.cc:492] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:01.125152 1158 ts_tablet_manager.cc:1397] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20250109 20:53:01.126258 2308 tablet.cc:1721] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:01.127035 1158 raft_consensus.cc:357] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.127223 1158 raft_consensus.cc:383] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.127271 1158 raft_consensus.cc:738] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.127498 1158 consensus_queue.cc:260] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.127931 1158 ts_tablet_manager.cc:1428] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 20:53:01.128084 746 tablet_service.cc:1467] Processing CreateTablet for tablet affb5b43f6314e08b0deb3b2e8b40207 (DEFAULT_TABLE table=default.Table_42be9454af31421f84024e4849874065 [id=efed9da4e675477dacd24addf44968dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.125727 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 7a5c88e044c64f638a7959b6dffea612 (DEFAULT_TABLE table=default.Table_7f04a87f321a4355bb2fc4cbfb0b9076 [id=e51ccdcd9fd543be913a7b884f112387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.129185 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3774aacd532242b788a7af6044dde6b3
I20250109 20:53:01.129456 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5c88e044c64f638a7959b6dffea612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.129881 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_16735ecbd9074c9ea64459d30539c2f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.130369 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16735ecbd9074c9ea64459d30539c2f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.131820 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 3774aacd532242b788a7af6044dde6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.132201 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 3774aacd532242b788a7af6044dde6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.133737 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5c88e044c64f638a7959b6dffea612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.135908 1030 tablet_service.cc:1467] Processing CreateTablet for tablet affb5b43f6314e08b0deb3b2e8b40207 (DEFAULT_TABLE table=default.Table_42be9454af31421f84024e4849874065 [id=efed9da4e675477dacd24addf44968dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.136497 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affb5b43f6314e08b0deb3b2e8b40207. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.138396 1165 tablet_bootstrap.cc:654] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.138747 892 tablet_service.cc:1467] Processing CreateTablet for tablet dfc973162d464694a4d59423182584ce (DEFAULT_TABLE table=default.Table_7598437489fd4408a5c7fafb1a722d66 [id=395e7eb3cfa4466eb459fa018787e29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.139572 2280 raft_consensus.cc:491] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.139688 2280 raft_consensus.cc:513] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.140259 2280 leader_election.cc:290] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:01.140362 745 tablet_service.cc:1467] Processing CreateTablet for tablet dfc973162d464694a4d59423182584ce (DEFAULT_TABLE table=default.Table_7598437489fd4408a5c7fafb1a722d66 [id=395e7eb3cfa4466eb459fa018787e29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.140473 2032 raft_consensus.cc:491] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.140592 2032 raft_consensus.cc:513] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.140962 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce0a941f0c14501a947a5c7e2a844e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:01.141160 2032 leader_election.cc:290] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:01.141487 989 leader_election.cc:343] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:01.145063 854 leader_election.cc:343] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8d1630e4438e4337ba026b41f630a480
I20250109 20:53:01.146143 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_38713c5cbbbf44ba95a5b4710cdfcd44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.146629 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38713c5cbbbf44ba95a5b4710cdfcd44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.152734 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cd9cd48910497ba3e4b159583cb7a6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:01.153090 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce0a941f0c14501a947a5c7e2a844e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:01.153555 990 leader_election.cc:343] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.153654 990 leader_election.cc:304] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:01.154000 990 leader_election.cc:343] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.154075 990 leader_election.cc:304] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:01.154625 2280 raft_consensus.cc:2743] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.154769 2280 raft_consensus.cc:2743] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.156924 895 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2cf264a78e4ff490050d7b014d936b (DEFAULT_TABLE table=default.Table_4784d2e8394f4928a758483708ee3504 [id=57fa697e23394e62ba1d5ec8ef7e5f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.157661 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5c88e044c64f638a7959b6dffea612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.160593 1032 tablet_service.cc:1467] Processing CreateTablet for tablet dfc973162d464694a4d59423182584ce (DEFAULT_TABLE table=default.Table_7598437489fd4408a5c7fafb1a722d66 [id=395e7eb3cfa4466eb459fa018787e29c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:01.161526 854 leader_election.cc:343] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 8d1630e4438e4337ba026b41f630a480
I20250109 20:53:01.161626 854 leader_election.cc:304] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.161835 2032 raft_consensus.cc:2743] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.165273 1166 ts_tablet_manager.cc:1918] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:01.166281 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc973162d464694a4d59423182584ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.168797 1166 log.cc:1198] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/881ea20c17904d669c8a1fbd766ba559
I20250109 20:53:01.173453 1160 ts_tablet_manager.cc:1918] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.173641 1160 log.cc:1198] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/af8c0bf27e994e6aa4cbedca16febe12
I20250109 20:53:01.174168 1160 ts_tablet_manager.cc:1939] T af8c0bf27e994e6aa4cbedca16febe12 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:01.174989 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet af8c0bf27e994e6aa4cbedca16febe12 (table default.Table_1a734623b43b4fab8389a393281e3a3f [id=3ecc13ed899846fe8cf09444c0c25504]) successfully deleted
I20250109 20:53:01.175457 1160 tablet_replica.cc:331] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:01.175793 1160 raft_consensus.cc:2238] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.176090 1160 raft_consensus.cc:2267] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.177433 1160 ts_tablet_manager.cc:1905] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.177986 1157 tablet_replica.cc:331] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.178167 1157 raft_consensus.cc:2238] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:01.178254 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 776e6e2b3b9a4f9f9d08361ba1cbb5ba (table default.Table_01d2dd63e9a4480b961c408dc7d9a0a9 [id=127a26e6b93f4873b9bab8bc378ac91a]) successfully deleted
I20250109 20:53:01.178435 1157 raft_consensus.cc:2267] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:01.178471 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0e4d5613811444c38ea786a6ad43f9c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e4d5613811444c38ea786a6ad43f9c1 already in progress: creating tablet
I20250109 20:53:01.178979 1157 ts_tablet_manager.cc:1905] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.179903 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 3774aacd532242b788a7af6044dde6b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.180378 747 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2cf264a78e4ff490050d7b014d936b (DEFAULT_TABLE table=default.Table_4784d2e8394f4928a758483708ee3504 [id=57fa697e23394e62ba1d5ec8ef7e5f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.180783 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2cf264a78e4ff490050d7b014d936b (DEFAULT_TABLE table=default.Table_4784d2e8394f4928a758483708ee3504 [id=57fa697e23394e62ba1d5ec8ef7e5f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.181120 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2cf264a78e4ff490050d7b014d936b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.181278 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2cf264a78e4ff490050d7b014d936b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.185634 1157 ts_tablet_manager.cc:1918] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:01.185786 1157 log.cc:1198] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/881ea20c17904d669c8a1fbd766ba559
I20250109 20:53:01.186252 1157 ts_tablet_manager.cc:1939] T 881ea20c17904d669c8a1fbd766ba559 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.186825 1157 tablet_replica.cc:331] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.186837 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4e29d5e170b040328770ef8b3a22d462 (DEFAULT_TABLE table=default.Table_75546481297c4d7f9038188240eb1852 [id=7362725042c042838bc4197e92556039]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.187026 1157 raft_consensus.cc:2238] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.187139 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e29d5e170b040328770ef8b3a22d462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.187170 1157 raft_consensus.cc:2267] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.187602 1157 ts_tablet_manager.cc:1905] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.188252 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_0504dfd389e44bafa7837757c1cec711 [id=3e073742d9cf476387b0a0949059d933]) successfully deleted
I20250109 20:53:01.188410 751 tablet_service.cc:1467] Processing CreateTablet for tablet f0634d1eb0064e9d85b83ea3506cc597 (DEFAULT_TABLE table=default.Table_045b529f170543478afcac3c58958866 [id=d65ebf45005448018d676483733e8c96]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:01.188472 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a1ee36f3be324b14868976298ca26118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ee36f3be324b14868976298ca26118 already in progress: creating tablet
I20250109 20:53:01.188789 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0634d1eb0064e9d85b83ea3506cc597. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.189079 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4b7e008a474978b91426f63dbd96ee (DEFAULT_TABLE table=default.Table_6d708581ee5c4179af45f5e8ce12bec5 [id=7a4b26558b3b43d097a1c01b0504b037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.189612 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affb5b43f6314e08b0deb3b2e8b40207. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.189754 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4b7e008a474978b91426f63dbd96ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.192703 887 tablet_service.cc:1467] Processing CreateTablet for tablet f0634d1eb0064e9d85b83ea3506cc597 (DEFAULT_TABLE table=default.Table_045b529f170543478afcac3c58958866 [id=d65ebf45005448018d676483733e8c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.192988 740 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4b7e008a474978b91426f63dbd96ee (DEFAULT_TABLE table=default.Table_6d708581ee5c4179af45f5e8ce12bec5 [id=7a4b26558b3b43d097a1c01b0504b037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.193272 883 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4b7e008a474978b91426f63dbd96ee (DEFAULT_TABLE table=default.Table_6d708581ee5c4179af45f5e8ce12bec5 [id=7a4b26558b3b43d097a1c01b0504b037]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.193346 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc973162d464694a4d59423182584ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.194607 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4e29d5e170b040328770ef8b3a22d462 (DEFAULT_TABLE table=default.Table_75546481297c4d7f9038188240eb1852 [id=7362725042c042838bc4197e92556039]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.195706 1158 tablet_bootstrap.cc:492] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:01.198038 1166 ts_tablet_manager.cc:1939] T 881ea20c17904d669c8a1fbd766ba559 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:01.198465 1158 tablet_bootstrap.cc:654] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.199455 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 881ea20c17904d669c8a1fbd766ba559 (table default.Table_0504dfd389e44bafa7837757c1cec711 [id=3e073742d9cf476387b0a0949059d933]) successfully deleted
I20250109 20:53:01.199477 1016 tablet_service.cc:1467] Processing CreateTablet for tablet f0634d1eb0064e9d85b83ea3506cc597 (DEFAULT_TABLE table=default.Table_045b529f170543478afcac3c58958866 [id=d65ebf45005448018d676483733e8c96]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:01.200636 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0e4d5613811444c38ea786a6ad43f9c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e4d5613811444c38ea786a6ad43f9c1 already in progress: creating tablet
I20250109 20:53:01.200929 1166 tablet_replica.cc:331] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:01.201107 1166 raft_consensus.cc:2238] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.201269 1166 raft_consensus.cc:2267] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.201638 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4b7e008a474978b91426f63dbd96ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.202011 1166 ts_tablet_manager.cc:1905] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.202515 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0634d1eb0064e9d85b83ea3506cc597. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.207481 747 tablet_service.cc:1467] Processing CreateTablet for tablet 4e29d5e170b040328770ef8b3a22d462 (DEFAULT_TABLE table=default.Table_75546481297c4d7f9038188240eb1852 [id=7362725042c042838bc4197e92556039]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.208076 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e29d5e170b040328770ef8b3a22d462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.209312 745 tablet_service.cc:1467] Processing CreateTablet for tablet d11e17f3391740b79ffa9d8a8c9bf400 (DEFAULT_TABLE table=default.Table_5fef44f0700046e0ba377025bb4d4c78 [id=96d115ee7e824e2eb0d56ff8787ea3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.209843 886 tablet_service.cc:1467] Processing CreateTablet for tablet d11e17f3391740b79ffa9d8a8c9bf400 (DEFAULT_TABLE table=default.Table_5fef44f0700046e0ba377025bb4d4c78 [id=96d115ee7e824e2eb0d56ff8787ea3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.210366 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11e17f3391740b79ffa9d8a8c9bf400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.215037 885 tablet_service.cc:1467] Processing CreateTablet for tablet e0f47f9b1f8a479a9760526a4873af57 (DEFAULT_TABLE table=default.Table_aa3f8eb194bf4a798c9efffc02e16f48 [id=f46bf4baee9643d8813926e3af2900de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.218235 1158 tablet_bootstrap.cc:492] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:01.218446 1158 ts_tablet_manager.cc:1397] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
W20250109 20:53:01.218991 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b087b7546a39451089666a36d029e645 consensus state has no leader
I20250109 20:53:01.220101 746 tablet_service.cc:1467] Processing CreateTablet for tablet e0f47f9b1f8a479a9760526a4873af57 (DEFAULT_TABLE table=default.Table_aa3f8eb194bf4a798c9efffc02e16f48 [id=f46bf4baee9643d8813926e3af2900de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.220396 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f47f9b1f8a479a9760526a4873af57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.222395 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d1574f6ea52544dd8cc0cec65fd89000" } new_table_name: "default.Table_ba16c6ffac0349ae83a5bdfd34a99fe5" modify_external_catalogs: true
I20250109 20:53:01.222647 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 with delete_type TABLET_DATA_DELETED (replaced by tablet beca5f1ed9d3434fb7de90cd74a6f09b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.221493 1032 tablet_service.cc:1467] Processing CreateTablet for tablet d11e17f3391740b79ffa9d8a8c9bf400 (DEFAULT_TABLE table=default.Table_5fef44f0700046e0ba377025bb4d4c78 [id=96d115ee7e824e2eb0d56ff8787ea3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.224342 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e0f47f9b1f8a479a9760526a4873af57 (DEFAULT_TABLE table=default.Table_aa3f8eb194bf4a798c9efffc02e16f48 [id=f46bf4baee9643d8813926e3af2900de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.225669 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet fb3e620799a943c6850e20a3d6d2f6bb
I20250109 20:53:01.226362 882 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3e620799a943c6850e20a3d6d2f6bb with delete_type TABLET_DATA_DELETED (replaced by tablet fb57395c2e684e878baf53d76802dc45) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.226619 451 catalog_manager.cc:6596] ReplaceTablet: tablet fb3e620799a943c6850e20a3d6d2f6bb deleted and replaced by tablet fb57395c2e684e878baf53d76802dc45
I20250109 20:53:01.226647 751 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3e620799a943c6850e20a3d6d2f6bb with delete_type TABLET_DATA_DELETED (replaced by tablet fb57395c2e684e878baf53d76802dc45) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.227253 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3e620799a943c6850e20a3d6d2f6bb with delete_type TABLET_DATA_DELETED (replaced by tablet fb57395c2e684e878baf53d76802dc45) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.227007 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_21ed68ee9ed84331ae175e7fe5d6bd6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.227948 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21ed68ee9ed84331ae175e7fe5d6bd6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.229019 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6a06112f354465912054990ba0ea58 (DEFAULT_TABLE table=default.Table_51c7449051f04e25bb0ee8eb814de9e1 [id=a31132be8d7d4219a7ea2a9de7b29260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.219604 1158 raft_consensus.cc:357] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.229197 1158 raft_consensus.cc:383] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.229328 1158 raft_consensus.cc:738] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.229341 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6a06112f354465912054990ba0ea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.229700 1158 consensus_queue.cc:260] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.231247 1158 ts_tablet_manager.cc:1428] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250109 20:53:01.233078 1158 tablet_bootstrap.cc:492] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:01.235270 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11e17f3391740b79ffa9d8a8c9bf400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.177517 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfbf64b178841d7aaf34632cbff3716. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.237752 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f47f9b1f8a479a9760526a4873af57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.237916 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4b7e008a474978b91426f63dbd96ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.238107 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2cf264a78e4ff490050d7b014d936b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.238301 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11e17f3391740b79ffa9d8a8c9bf400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.238569 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0634d1eb0064e9d85b83ea3506cc597. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.239652 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc973162d464694a4d59423182584ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.241564 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e29d5e170b040328770ef8b3a22d462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.243343 1165 tablet_bootstrap.cc:492] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:01.243461 1165 ts_tablet_manager.cc:1397] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.119s user 0.001s sys 0.001s
I20250109 20:53:01.244004 1157 ts_tablet_manager.cc:1918] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.246834 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6a06112f354465912054990ba0ea58 (DEFAULT_TABLE table=default.Table_51c7449051f04e25bb0ee8eb814de9e1 [id=a31132be8d7d4219a7ea2a9de7b29260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.246832 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6a06112f354465912054990ba0ea58 (DEFAULT_TABLE table=default.Table_51c7449051f04e25bb0ee8eb814de9e1 [id=a31132be8d7d4219a7ea2a9de7b29260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.247148 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6a06112f354465912054990ba0ea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.248441 1165 raft_consensus.cc:357] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.248710 1165 raft_consensus.cc:383] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.248859 1165 raft_consensus.cc:738] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.251286 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6a06112f354465912054990ba0ea58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.253103 893 tablet_service.cc:1467] Processing CreateTablet for tablet 340d2633f6c748498f6f60d5ad156793 (DEFAULT_TABLE table=default.Table_d9f9c4769d054758bcfc87654ec86bfc [id=0140b301d7de47ceb6775d35a4867842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.253388 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340d2633f6c748498f6f60d5ad156793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.256821 1157 log.cc:1198] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ffd3a1b1d3e44feda6dbaa8556e377fc
I20250109 20:53:01.256839 1165 consensus_queue.cc:260] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.252498 1158 tablet_bootstrap.cc:654] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.258111 1165 ts_tablet_manager.cc:1428] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20250109 20:53:01.258590 1165 tablet_bootstrap.cc:492] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:01.260782 1157 ts_tablet_manager.cc:1939] T ffd3a1b1d3e44feda6dbaa8556e377fc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.261570 1166 ts_tablet_manager.cc:1918] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.263163 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ffd3a1b1d3e44feda6dbaa8556e377fc (table default.Table_b856f34058994714afb4ce41d44f5384 [id=7adda20a8381463a9f6decd4c7cd045d]) successfully deleted
I20250109 20:53:01.267612 1166 log.cc:1198] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f03d4e679a1c4793bd83cd979acced88
I20250109 20:53:01.268218 1166 ts_tablet_manager.cc:1939] T f03d4e679a1c4793bd83cd979acced88 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:01.269367 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f47f9b1f8a479a9760526a4873af57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.269382 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f03d4e679a1c4793bd83cd979acced88 (table default.Table_7592f0da4beb49cda4e5e9c6392a085b [id=5fbc12ffb04546deb78133a5796c63c6]) successfully deleted
I20250109 20:53:01.269801 1157 tablet_replica.cc:331] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.270085 1157 raft_consensus.cc:2238] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.270299 1157 raft_consensus.cc:2267] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:01.272262 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b6bcf34f4788483783acb3d09accba37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6bcf34f4788483783acb3d09accba37 already in progress: creating tablet
I20250109 20:53:01.273442 1157 ts_tablet_manager.cc:1905] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:01.274384 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 01174f43d56c4c96a3b754828fc30640 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01174f43d56c4c96a3b754828fc30640 already in progress: creating tablet
I20250109 20:53:01.281383 742 tablet_service.cc:1467] Processing CreateTablet for tablet 340d2633f6c748498f6f60d5ad156793 (DEFAULT_TABLE table=default.Table_d9f9c4769d054758bcfc87654ec86bfc [id=0140b301d7de47ceb6775d35a4867842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.281998 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340d2633f6c748498f6f60d5ad156793. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:01.282158 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3af9563f6efd47059caac7c86c41d43f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3af9563f6efd47059caac7c86c41d43f already in progress: creating tablet
W20250109 20:53:01.285240 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a6006590b7764efc901bf51ca4e19ab3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6006590b7764efc901bf51ca4e19ab3 already in progress: creating tablet
I20250109 20:53:01.286825 2253 consensus_queue.cc:1035] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:53:01.287038 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 11dbbc9ceb1a4252b1605560bdb9b62a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11dbbc9ceb1a4252b1605560bdb9b62a already in progress: creating tablet
I20250109 20:53:01.287768 1158 tablet_bootstrap.cc:492] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:01.287868 1158 ts_tablet_manager.cc:1397] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20250109 20:53:01.288550 1158 raft_consensus.cc:357] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.288767 1158 raft_consensus.cc:383] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.288815 1158 raft_consensus.cc:738] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.288961 1158 consensus_queue.cc:260] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.289412 1158 ts_tablet_manager.cc:1428] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250109 20:53:01.291527 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0ccc75429f7441f1b186879bc5d12f92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ccc75429f7441f1b186879bc5d12f92 already in progress: creating tablet
I20250109 20:53:01.291754 1166 tablet_replica.cc:331] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:01.291918 1166 raft_consensus.cc:2238] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.292032 1166 raft_consensus.cc:2267] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.294040 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 340d2633f6c748498f6f60d5ad156793 (DEFAULT_TABLE table=default.Table_d9f9c4769d054758bcfc87654ec86bfc [id=0140b301d7de47ceb6775d35a4867842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.294377 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340d2633f6c748498f6f60d5ad156793. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:01.301393 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb562bcef6c44710a657db7631ac1196 consensus state has no leader
I20250109 20:53:01.303385 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ee5424d3946544f49aa8a1f94a0fbbc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.304013 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee5424d3946544f49aa8a1f94a0fbbc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.304381 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]). Expected version 3 got 0
I20250109 20:53:01.304562 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]). Expected version 1 got 0
W20250109 20:53:01.305521 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50afaf6c42a546b2ad7add93095a7556 consensus state has no leader
I20250109 20:53:01.305734 452 master_service.cc:946] ReplaceTablet: received request to replace tablet d4b939cdea1c4719b9b3918aabbdfc0e from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:01.306986 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fc79e5e4f5114b189904f056b147d163" } modify_external_catalogs: true
I20250109 20:53:01.307121 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fc79e5e4f5114b189904f056b147d163" } modify_external_catalogs: true
I20250109 20:53:01.308262 1166 ts_tablet_manager.cc:1905] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.309355 1176 tablet_bootstrap.cc:492] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.310261 1176 ts_tablet_manager.cc:1397] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.233s user 0.011s sys 0.031s
I20250109 20:53:01.308703 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_84dfe41ea9994c1aa62b04e2ec77e4c9" } modify_external_catalogs: true
I20250109 20:53:01.310986 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_84dfe41ea9994c1aa62b04e2ec77e4c9" } modify_external_catalogs: true
I20250109 20:53:01.314026 1176 raft_consensus.cc:357] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.314780 1176 raft_consensus.cc:383] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.316689 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d2ad301c11943b69b59be18d6775513 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b1822cd5744b5197d8141fc66539a0) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.317832 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:01.318213 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:53:01.325026 2253 raft_consensus.cc:491] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.325161 2253 raft_consensus.cc:513] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.325719 2253 leader_election.cc:290] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:01.326514 713 leader_election.cc:343] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 728e9b6101e3447fb3c24b3917aa4872
I20250109 20:53:01.319875 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dc6847928fe0476aae2e436f966abd99" } modify_external_catalogs: true
I20250109 20:53:01.319993 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_142271ba6e494ddd876859927b650918" } modify_external_catalogs: true
I20250109 20:53:01.327399 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_142271ba6e494ddd876859927b650918" } modify_external_catalogs: true
I20250109 20:53:01.327582 1157 ts_tablet_manager.cc:1918] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.327664 1157 log.cc:1198] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/35a39fac67894a208dd7ec3153b05e4a
W20250109 20:53:01.323740 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d0e0f4e3a82a437ebbb16426fffa5e2f (table default.Table_0192f16726024e458eb0eed5b1b5dfd2 [id=51a58f4a2e5e4fb3890abd11e22e8110], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d0e0f4e3a82a437ebbb16426fffa5e2f consensus state has no leader
W20250109 20:53:01.324626 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ab297e2c76349cb97273200f6a950d8 consensus state has no leader
I20250109 20:53:01.325042 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2fdd49df493a4c26bcdea27254701757" } new_table_name: "default.Table_8ca6e16543a94e608f9e3a73747ec6d2" modify_external_catalogs: true
I20250109 20:53:01.327919 1157 ts_tablet_manager.cc:1939] T 35a39fac67894a208dd7ec3153b05e4a P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.327862 1176 raft_consensus.cc:738] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.325677 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_08bc5abe888e442ba8afb7fe0b9d746a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.328800 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 35a39fac67894a208dd7ec3153b05e4a (table default.Table_df4e58889f9d4ef6b81876c48ecf223e [id=fcb76535fbbe44e29ef627a3ae562ce2]) successfully deleted
W20250109 20:53:01.329113 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08bc5abe888e442ba8afb7fe0b9d746a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:01.329666 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8190cb332af34a398ae8970567562d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8190cb332af34a398ae8970567562d0e already in progress: creating tablet
I20250109 20:53:01.326058 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d4b939cdea1c4719b9b3918aabbdfc0e
I20250109 20:53:01.330750 452 catalog_manager.cc:6596] ReplaceTablet: tablet d4b939cdea1c4719b9b3918aabbdfc0e deleted and replaced by tablet e6ab69a3204a4c288b369b5319b1ee0d
I20250109 20:53:01.326792 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 062f1d04adc14599ace583896a3117e0
I20250109 20:53:01.327016 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 67be9533983443cbb8fc52031216a973
I20250109 20:53:01.327242 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dc6847928fe0476aae2e436f966abd99" } modify_external_catalogs: true
I20250109 20:53:01.333021 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b939cdea1c4719b9b3918aabbdfc0e with delete_type TABLET_DATA_DELETED (replaced by tablet e6ab69a3204a4c288b369b5319b1ee0d) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.331210 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b939cdea1c4719b9b3918aabbdfc0e with delete_type TABLET_DATA_DELETED (replaced by tablet e6ab69a3204a4c288b369b5319b1ee0d) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.334755 1157 tablet_replica.cc:331] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.334910 1157 raft_consensus.cc:2238] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.335067 1157 raft_consensus.cc:2267] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.335538 1157 ts_tablet_manager.cc:1905] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.328962 1176 consensus_queue.cc:260] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.336385 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 67be9533983443cbb8fc52031216a973 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.338004 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:01.338387 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1340076b6d8146429e90b485b3b58728" } modify_external_catalogs: true
I20250109 20:53:01.338557 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1340076b6d8146429e90b485b3b58728" } modify_external_catalogs: true
I20250109 20:53:01.339851 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 67be9533983443cbb8fc52031216a973 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.340075 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 062f1d04adc14599ace583896a3117e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.340883 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9f4e74a5b52948adb470c7bbc4616b49
I20250109 20:53:01.342376 1160 ts_tablet_manager.cc:1918] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.342482 1160 log.cc:1198] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/8d1630e4438e4337ba026b41f630a480
I20250109 20:53:01.342764 1160 ts_tablet_manager.cc:1939] T 8d1630e4438e4337ba026b41f630a480 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:01.344748 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4e74a5b52948adb470c7bbc4616b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.345336 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728e9b6101e3447fb3c24b3917aa4872" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:01.345847 712 leader_election.cc:343] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.345959 712 leader_election.cc:304] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.346307 2253 raft_consensus.cc:2743] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.348363 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7a0bfb44367a4572a9507af0448af846
I20250109 20:53:01.349764 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0bfb44367a4572a9507af0448af846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.350003 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0bfb44367a4572a9507af0448af846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.350167 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_62fca2bdab804af8ab92b2eb60115336"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.350639 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62fca2bdab804af8ab92b2eb60115336 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.351979 1160 tablet_replica.cc:331] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:01.352169 1160 raft_consensus.cc:2238] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.352298 1160 raft_consensus.cc:2267] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.352906 1160 ts_tablet_manager.cc:1905] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.353299 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4e74a5b52948adb470c7bbc4616b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.355103 2253 raft_consensus.cc:491] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.355227 2253 raft_consensus.cc:513] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.355367 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 62caa775ef1241f68c0323a2d228c8cd
I20250109 20:53:01.355820 2253 leader_election.cc:290] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.356412 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 8d1630e4438e4337ba026b41f630a480 (table default.Table_d7539ac3866842c996d4ced88e764877 [id=b78805ca41e647e1927d2564fb00b21f]) successfully deleted
W20250109 20:53:01.356601 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 01174f43d56c4c96a3b754828fc30640 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01174f43d56c4c96a3b754828fc30640 already in progress: creating tablet
W20250109 20:53:01.356839 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3af9563f6efd47059caac7c86c41d43f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3af9563f6efd47059caac7c86c41d43f already in progress: creating tablet
W20250109 20:53:01.357009 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a6006590b7764efc901bf51ca4e19ab3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6006590b7764efc901bf51ca4e19ab3 already in progress: creating tablet
W20250109 20:53:01.357180 713 leader_election.cc:343] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 57e56f30e6f3450a8426cd40db89e009
W20250109 20:53:01.357203 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 11dbbc9ceb1a4252b1605560bdb9b62a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11dbbc9ceb1a4252b1605560bdb9b62a already in progress: creating tablet
W20250109 20:53:01.357367 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d40b8ce042f349d599ce2a3bd2e4a91d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d40b8ce042f349d599ce2a3bd2e4a91d already in progress: creating tablet
I20250109 20:53:01.357508 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 62caa775ef1241f68c0323a2d228c8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:01.357586 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0ccc75429f7441f1b186879bc5d12f92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ccc75429f7441f1b186879bc5d12f92 already in progress: creating tablet
I20250109 20:53:01.358908 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 62caa775ef1241f68c0323a2d228c8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.359833 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 062f1d04adc14599ace583896a3117e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.360150 1158 tablet_bootstrap.cc:492] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:01.361654 1158 tablet_bootstrap.cc:654] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.365216 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b939cdea1c4719b9b3918aabbdfc0e with delete_type TABLET_DATA_DELETED (replaced by tablet e6ab69a3204a4c288b369b5319b1ee0d) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.365798 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b with delete_type TABLET_DATA_DELETED (replaced by tablet 50039da210df47ab8476e28387c8a802) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.375594 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e56f30e6f3450a8426cd40db89e009 with delete_type TABLET_DATA_DELETED (replaced by tablet eaea4f239cce41a997496a1c94810ac9) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.375723 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d409409813420689134773c627833b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.379272 1157 ts_tablet_manager.cc:1918] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.380966 1165 tablet_bootstrap.cc:654] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.382045 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e56f30e6f3450a8426cd40db89e009" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:01.383548 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4e74a5b52948adb470c7bbc4616b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:01.383064 712 leader_election.cc:343] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.385978 712 leader_election.cc:304] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.386328 1984 raft_consensus.cc:2743] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.385295 1158 tablet_bootstrap.cc:492] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:01.386636 1158 ts_tablet_manager.cc:1397] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.002s
I20250109 20:53:01.386775 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 62caa775ef1241f68c0323a2d228c8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.386927 919 raft_consensus.cc:3054] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:01.387161 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 67be9533983443cbb8fc52031216a973 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.387332 1158 raft_consensus.cc:357] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.387580 1158 raft_consensus.cc:383] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.387670 1158 raft_consensus.cc:738] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.387902 1158 consensus_queue.cc:260] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.392520 1158 ts_tablet_manager.cc:1428] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:53:01.392815 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0bfb44367a4572a9507af0448af846 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.397424 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da1f3cafffe4f7ca39624b43831689e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.397724 1158 tablet_bootstrap.cc:492] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:01.398147 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 062f1d04adc14599ace583896a3117e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.400660 1158 tablet_bootstrap.cc:654] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.401746 1176 ts_tablet_manager.cc:1428] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.091s user 0.002s sys 0.000s
I20250109 20:53:01.403920 1157 log.cc:1198] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/9558dfe563d94482bf496e19b8a80edc
I20250109 20:53:01.404227 1157 ts_tablet_manager.cc:1939] T 9558dfe563d94482bf496e19b8a80edc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.407116 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 9558dfe563d94482bf496e19b8a80edc (table default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6]) successfully deleted
W20250109 20:53:01.408072 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b6bcf34f4788483783acb3d09accba37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6bcf34f4788483783acb3d09accba37 already in progress: creating tablet
I20250109 20:53:01.408553 1157 tablet_replica.cc:331] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.408859 1157 raft_consensus.cc:2238] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:01.400432 2321 log.cc:926] Time spent T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20250109 20:53:01.410130 712 consensus_peers.cc:487] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:01.413887 1157 raft_consensus.cc:2267] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.414310 1157 ts_tablet_manager.cc:1905] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:01.417735 440 catalog_manager.cc:4945] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): alter failed for tablet c00aaf81b34445abb7502700f01d4f3f (table default.Table_385822fe7219471bacdd1ad2dbe7d891 [id=0430588a51ed49ae86290eaab54f79dc]),no further retry: Not found: Tablet not found: c00aaf81b34445abb7502700f01d4f3f
W20250109 20:53:01.421499 712 consensus_peers.cc:487] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:01.423590 1984 raft_consensus.cc:491] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.423696 1984 raft_consensus.cc:513] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.424249 1984 leader_election.cc:290] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.425288 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daac99da5d10475c9de263bef27baeb7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:01.425469 1052 raft_consensus.cc:2463] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:01.427484 713 leader_election.cc:304] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:01.428447 1984 raft_consensus.cc:2798] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:01.428517 1984 raft_consensus.cc:491] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:01.428570 1984 raft_consensus.cc:3054] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:01.430374 1160 ts_tablet_manager.cc:1918] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.430855 1984 raft_consensus.cc:513] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.431907 2254 consensus_queue.cc:1035] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:01.432408 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daac99da5d10475c9de263bef27baeb7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:01.432518 1984 leader_election.cc:290] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.432579 1052 raft_consensus.cc:3054] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:01.435124 1052 raft_consensus.cc:2463] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:01.435568 713 leader_election.cc:304] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:01.435885 1984 raft_consensus.cc:2798] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:01.436087 1984 raft_consensus.cc:695] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:01.436296 1984 consensus_queue.cc:237] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.438179 446 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:01.432905 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3f42577ce23e4ed6b0f6ef3d41a6f5d0" } new_table_name: "default.Table_06b47da1327e425989c115839e9aecbe" modify_external_catalogs: true
I20250109 20:53:01.443827 2318 raft_consensus.cc:491] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.443946 2318 raft_consensus.cc:513] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.444537 2318 leader_election.cc:290] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:01.445699 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed7abf23ce04a268a460d8e3e97d56b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:01.432371 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daac99da5d10475c9de263bef27baeb7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:01.446363 990 leader_election.cc:343] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.446559 1160 log.cc:1198] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/34f215eee55d4e6a97a384df6ff9d596
W20250109 20:53:01.446738 712 leader_election.cc:343] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.435405 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daac99da5d10475c9de263bef27baeb7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:01.445070 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed7abf23ce04a268a460d8e3e97d56b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:01.447575 712 leader_election.cc:343] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:01.447575 989 leader_election.cc:343] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.447654 989 leader_election.cc:304] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:01.446869 1160 ts_tablet_manager.cc:1939] T 34f215eee55d4e6a97a384df6ff9d596 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:01.447944 2318 raft_consensus.cc:2743] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.448412 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 34f215eee55d4e6a97a384df6ff9d596 (table default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]) successfully deleted
I20250109 20:53:01.448715 1160 tablet_replica.cc:331] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:01.448907 1160 raft_consensus.cc:2238] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.449038 1160 raft_consensus.cc:2267] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:01.448412 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82209a6b39bb48e791ff63a8167c9765 consensus state has no leader
I20250109 20:53:01.450752 1160 ts_tablet_manager.cc:1905] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.452945 1165 tablet_bootstrap.cc:492] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:01.453188 1165 ts_tablet_manager.cc:1397] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.195s user 0.003s sys 0.005s
I20250109 20:53:01.453886 1165 raft_consensus.cc:357] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.454109 1165 raft_consensus.cc:383] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.454161 1165 raft_consensus.cc:738] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.461099 1165 consensus_queue.cc:260] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.461777 1165 ts_tablet_manager.cc:1428] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250109 20:53:01.462018 1165 tablet_bootstrap.cc:492] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:01.461905 1176 tablet_bootstrap.cc:492] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:01.463160 1165 tablet_bootstrap.cc:654] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.464000 1176 tablet_bootstrap.cc:654] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.465358 1157 ts_tablet_manager.cc:1918] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.465476 1157 log.cc:1198] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2a4f047e8fa34ca79e78915c61a8ba12
I20250109 20:53:01.465862 1157 ts_tablet_manager.cc:1939] T 2a4f047e8fa34ca79e78915c61a8ba12 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.466770 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2a4f047e8fa34ca79e78915c61a8ba12 (table default.Table_70499a7567fa4d6498705309cde3b257 [id=3ae741f5fe4343188b6f0d10b2a09b69]) successfully deleted
I20250109 20:53:01.480228 1160 ts_tablet_manager.cc:1918] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.481576 1157 tablet_replica.cc:331] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.482012 1157 raft_consensus.cc:2238] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.482347 1160 log.cc:1198] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ed5ea5d588834f1b8b2b057ec324aadc
I20250109 20:53:01.482559 1157 raft_consensus.cc:2267] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.483007 1160 ts_tablet_manager.cc:1939] T ed5ea5d588834f1b8b2b057ec324aadc P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:01.487936 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ed5ea5d588834f1b8b2b057ec324aadc (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:53:01.484522 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 47726b58d6714cd1b46848a16a34ff37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.488566 1157 ts_tablet_manager.cc:1905] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.488837 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
W20250109 20:53:01.489336 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3774aacd532242b788a7af6044dde6b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3774aacd532242b788a7af6044dde6b3 already in progress: creating tablet
W20250109 20:53:01.492622 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet fb3e620799a943c6850e20a3d6d2f6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb3e620799a943c6850e20a3d6d2f6bb already in progress: creating tablet
I20250109 20:53:01.493202 1160 tablet_replica.cc:331] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:01.493592 1160 raft_consensus.cc:2238] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.493964 1160 raft_consensus.cc:2267] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.494503 2308 tablet.cc:1721] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:01.499281 1157 ts_tablet_manager.cc:1918] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.500356 446 catalog_manager.cc:5526] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:01.505677 888 tablet_service.cc:1467] Processing CreateTablet for tablet 05df026727da4ea594810bb9b516dc7c (DEFAULT_TABLE table=default.Table_99e81a65fa2741ac80eccc081148192a [id=ddd3c972257b48878e250cf287622605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.506522 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05df026727da4ea594810bb9b516dc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.510406 1166 ts_tablet_manager.cc:1918] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.510591 1166 log.cc:1198] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/28cd9cd48910497ba3e4b159583cb7a6
I20250109 20:53:01.511193 1166 ts_tablet_manager.cc:1939] T 28cd9cd48910497ba3e4b159583cb7a6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:01.511714 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e46707392eee49959bca8c498cf57a76" } new_table_name: "default.Table_020e53b9d9e64e23b63d341a15c2ff78" modify_external_catalogs: true
I20250109 20:53:01.511917 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_08105d5a57484ec6b34290b8daa1e416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.512428 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08105d5a57484ec6b34290b8daa1e416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.513069 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_75f8ca17cad34737a386cc58f15a0697" } new_table_name: "default.Table_b28d493028f4488394c6d101b7b917e6" modify_external_catalogs: true
I20250109 20:53:01.513577 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2ebffe648ce34afaa688cf8cb881e20f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.514009 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ebffe648ce34afaa688cf8cb881e20f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.515928 1160 ts_tablet_manager.cc:1905] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.517612 1176 tablet_bootstrap.cc:492] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.517769 1176 ts_tablet_manager.cc:1397] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.001s
I20250109 20:53:01.518486 1176 raft_consensus.cc:357] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.521997 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:53:01.522943 1176 raft_consensus.cc:383] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.523090 1165 tablet_bootstrap.cc:492] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:01.523090 1176 raft_consensus.cc:738] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.523394 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9e2fb7942ca744539c839f031e3528ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.523540 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a695f0ab86f145438002bbe953aad670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.523702 1176 consensus_queue.cc:260] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:01.523921 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a695f0ab86f145438002bbe953aad670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.524246 1176 ts_tablet_manager.cc:1428] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:53:01.524492 1176 tablet_bootstrap.cc:492] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:53:01.529335 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a8e0351fe654fa68e31e4308beb1c79 consensus state has no leader
W20250109 20:53:01.523922 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e2fb7942ca744539c839f031e3528ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.529858 1176 tablet_bootstrap.cc:654] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.531005 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8aa7a8f99b5a4c5293df93fcbd7ba167"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.531556 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8aa7a8f99b5a4c5293df93fcbd7ba167 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.531831 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6da7024fb0764c07a7d8ef603e5196f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.532337 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6da7024fb0764c07a7d8ef603e5196f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:01.536422 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edfcbc74f489406297bfa8df228d2787 consensus state has no leader
I20250109 20:53:01.537065 1165 ts_tablet_manager.cc:1397] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.002s
I20250109 20:53:01.537168 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 28cd9cd48910497ba3e4b159583cb7a6 (table default.Table_49bb1f7179c6420c83e530237052e180 [id=38d4e8c8c6294d3983708d35f536e400]) successfully deleted
W20250109 20:53:01.539018 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d40b8ce042f349d599ce2a3bd2e4a91d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d40b8ce042f349d599ce2a3bd2e4a91d already in progress: creating tablet
I20250109 20:53:01.539727 1165 raft_consensus.cc:357] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.539928 1165 raft_consensus.cc:383] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.539994 1165 raft_consensus.cc:738] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.500890 747 tablet_service.cc:1467] Processing CreateTablet for tablet 05df026727da4ea594810bb9b516dc7c (DEFAULT_TABLE table=default.Table_99e81a65fa2741ac80eccc081148192a [id=ddd3c972257b48878e250cf287622605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.540882 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05df026727da4ea594810bb9b516dc7c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:01.541208 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3774aacd532242b788a7af6044dde6b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3774aacd532242b788a7af6044dde6b3 already in progress: creating tablet
I20250109 20:53:01.541417 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4453facc7e5d4f2f8f7cacf989aff0fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.541851 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4453facc7e5d4f2f8f7cacf989aff0fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:01.542758 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet fb3e620799a943c6850e20a3d6d2f6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb3e620799a943c6850e20a3d6d2f6bb already in progress: creating tablet
I20250109 20:53:01.542766 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7147fbe7700d4c52a663ea8345c0a31d" } new_table_name: "default.Table_327d509e9b2847b496c172b6c277d1c3" modify_external_catalogs: true
I20250109 20:53:01.543984 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c18d342c6c2f416e88d8e3049d4909f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.540679 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
W20250109 20:53:01.544432 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c18d342c6c2f416e88d8e3049d4909f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:01.501607 712 consensus_peers.cc:487] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:01.544778 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_54e2dbe4b2ac420ba5f1ebd060213e74" } modify_external_catalogs: true
I20250109 20:53:01.544930 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_54e2dbe4b2ac420ba5f1ebd060213e74" } modify_external_catalogs: true
I20250109 20:53:01.545912 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:01.547534 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_8ca6e16543a94e608f9e3a73747ec6d2 [id=2ab3ba8731124704889904608a2f0cd3] alter complete (version 3)
I20250109 20:53:01.547888 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d70feebb4775482eb8b3b251c50ad92c" } modify_external_catalogs: true
I20250109 20:53:01.548018 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d70feebb4775482eb8b3b251c50ad92c" } modify_external_catalogs: true
I20250109 20:53:01.509243 1158 tablet_bootstrap.cc:492] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
W20250109 20:53:01.548173 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d4b939cdea1c4719b9b3918aabbdfc0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b939cdea1c4719b9b3918aabbdfc0e already in progress: creating tablet
I20250109 20:53:01.548213 1158 ts_tablet_manager.cc:1397] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.151s user 0.008s sys 0.003s
I20250109 20:53:01.549242 1158 raft_consensus.cc:357] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.549469 1158 raft_consensus.cc:383] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.549568 1158 raft_consensus.cc:738] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:01.550115 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 67be9533983443cbb8fc52031216a973 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67be9533983443cbb8fc52031216a973 already in progress: creating tablet
I20250109 20:53:01.550151 1158 consensus_queue.cc:260] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.550704 1158 ts_tablet_manager.cc:1428] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:01.552141 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:53:01.552311 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 5f953f43877c4a738ffd8255afbf8275 (table default.Table_3254b432932b4d9bb83cb8a2f26e5514 [id=fa8fa63dbd44442b81a3b9640d42680e]). Expected version 1 got 0
I20250109 20:53:01.552484 1158 tablet_bootstrap.cc:492] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:01.553031 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9f4e74a5b52948adb470c7bbc4616b49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f4e74a5b52948adb470c7bbc4616b49 already in progress: creating tablet
I20250109 20:53:01.553892 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_334eff3522a142c4b245128efa66f4c6" } new_table_name: "default.Table_3ca698d964314e04adb337451d78279c" modify_external_catalogs: true
I20250109 20:53:01.554082 1158 tablet_bootstrap.cc:654] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:01.554886 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7a0bfb44367a4572a9507af0448af846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a0bfb44367a4572a9507af0448af846 already in progress: creating tablet
I20250109 20:53:01.555382 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6223d4bbeb7a4deebddb2936e92224c0" } modify_external_catalogs: true
I20250109 20:53:01.555691 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6223d4bbeb7a4deebddb2936e92224c0" } modify_external_catalogs: true
I20250109 20:53:01.556931 1166 tablet_replica.cc:331] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:01.558121 1157 log.cc:1198] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/728e9b6101e3447fb3c24b3917aa4872
I20250109 20:53:01.558385 2318 raft_consensus.cc:491] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.558490 2318 raft_consensus.cc:513] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.558683 1157 ts_tablet_manager.cc:1939] T 728e9b6101e3447fb3c24b3917aa4872 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.559054 2318 leader_election.cc:290] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:01.559274 2343 tablet.cc:1721] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:01.559785 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f953f43877c4a738ffd8255afbf8275" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:01.560175 990 leader_election.cc:343] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.560506 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 728e9b6101e3447fb3c24b3917aa4872 (table default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]) successfully deleted
W20250109 20:53:01.560694 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 01174f43d56c4c96a3b754828fc30640 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01174f43d56c4c96a3b754828fc30640 already in progress: creating tablet
W20250109 20:53:01.560926 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0ccc75429f7441f1b186879bc5d12f92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ccc75429f7441f1b186879bc5d12f92 already in progress: creating tablet
W20250109 20:53:01.561108 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 11dbbc9ceb1a4252b1605560bdb9b62a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11dbbc9ceb1a4252b1605560bdb9b62a already in progress: creating tablet
I20250109 20:53:01.561551 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f953f43877c4a738ffd8255afbf8275" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:01.561681 1176 tablet_bootstrap.cc:492] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.561794 1176 ts_tablet_manager.cc:1397] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
W20250109 20:53:01.562095 989 leader_election.cc:343] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.562181 989 leader_election.cc:304] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:01.562626 2318 raft_consensus.cc:2743] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:01.562692 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3af9563f6efd47059caac7c86c41d43f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3af9563f6efd47059caac7c86c41d43f already in progress: creating tablet
I20250109 20:53:01.562788 1165 consensus_queue.cc:260] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:01.563160 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a6006590b7764efc901bf51ca4e19ab3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6006590b7764efc901bf51ca4e19ab3 already in progress: creating tablet
W20250109 20:53:01.563715 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d40b8ce042f349d599ce2a3bd2e4a91d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d40b8ce042f349d599ce2a3bd2e4a91d already in progress: creating tablet
I20250109 20:53:01.563515 1176 raft_consensus.cc:357] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.564029 1176 raft_consensus.cc:383] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.564085 1176 raft_consensus.cc:738] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.564272 1176 consensus_queue.cc:260] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:01.564821 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3774aacd532242b788a7af6044dde6b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3774aacd532242b788a7af6044dde6b3 already in progress: creating tablet
I20250109 20:53:01.566221 1157 tablet_replica.cc:331] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.566430 1157 raft_consensus.cc:2238] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.566602 1157 raft_consensus.cc:2267] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.566992 2340 raft_consensus.cc:491] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.567099 2340 raft_consensus.cc:513] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.567638 2340 leader_election.cc:290] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.568629 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:01.570369 1176 ts_tablet_manager.cc:1428] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:53:01.570641 1176 tablet_bootstrap.cc:492] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:01.571182 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da1f3cafffe4f7ca39624b43831689e" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:01.571631 712 leader_election.cc:343] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.571887 1176 tablet_bootstrap.cc:654] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.572305 1165 ts_tablet_manager.cc:1428] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
W20250109 20:53:01.572441 713 leader_election.cc:343] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 7da1f3cafffe4f7ca39624b43831689e
I20250109 20:53:01.572530 713 leader_election.cc:304] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.574394 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_31e530948ce441d0bfa57b94cd6df55d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.574719 2340 raft_consensus.cc:2743] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:01.574839 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31e530948ce441d0bfa57b94cd6df55d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.575336 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e46f4312844b6eb29d6718f4800f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.576349 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dbccae0af24a629600a92e7eadfee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:45 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.579912 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a8bbb3da84088a98c816102e4e9df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.581300 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bbb3e86398ca4302b3f758d9a36837f9" } modify_external_catalogs: true
I20250109 20:53:01.581492 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bbb3e86398ca4302b3f758d9a36837f9" } modify_external_catalogs: true
I20250109 20:53:01.581476 1166 raft_consensus.cc:2238] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.582690 1157 ts_tablet_manager.cc:1905] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.583179 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9939ee8672474bc0bcd631684079d0c3" } modify_external_catalogs: true
I20250109 20:53:01.583169 1166 raft_consensus.cc:2267] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.583397 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9939ee8672474bc0bcd631684079d0c3" } modify_external_catalogs: true
I20250109 20:53:01.584295 1165 tablet_bootstrap.cc:492] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:01.586707 2318 raft_consensus.cc:491] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.586815 2318 raft_consensus.cc:513] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.586988 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8d8220072e9f42ef90eda39e2382d2df
I20250109 20:53:01.587293 2318 leader_election.cc:290] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:01.587626 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8220072e9f42ef90eda39e2382d2df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.587852 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3c276fab6a4d068d64ea5d23c9a595" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:01.587872 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3c276fab6a4d068d64ea5d23c9a595" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:01.588135 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b263bd08e0584027b3c4534c8d423710"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.588267 989 leader_election.cc:343] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:01.588415 990 leader_election.cc:343] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.588474 990 leader_election.cc:304] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:01.588579 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b263bd08e0584027b3c4534c8d423710 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.588706 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8220072e9f42ef90eda39e2382d2df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.588994 2318 raft_consensus.cc:2743] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.589468 1158 tablet_bootstrap.cc:492] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:01.590163 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet daac99da5d10475c9de263bef27baeb7
I20250109 20:53:01.590799 747 tablet_service.cc:1514] Processing DeleteTablet for tablet daac99da5d10475c9de263bef27baeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.591380 888 tablet_service.cc:1514] Processing DeleteTablet for tablet daac99da5d10475c9de263bef27baeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:01.591678 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6fa1245ccea427eb6b2c0eaadebd24e consensus state has no leader
I20250109 20:53:01.592386 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet daac99da5d10475c9de263bef27baeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.592828 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8220072e9f42ef90eda39e2382d2df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.594771 1166 ts_tablet_manager.cc:1905] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.595494 1165 tablet_bootstrap.cc:654] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.595922 1157 ts_tablet_manager.cc:1918] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.597501 1176 tablet_bootstrap.cc:492] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.597689 1176 ts_tablet_manager.cc:1397] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
W20250109 20:53:01.597903 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e392ffa3384491da72da4b713e4a132 consensus state has no leader
I20250109 20:53:01.598311 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8f77f35085ff43a5b7fb6cce7e7a3409
I20250109 20:53:01.598875 1176 raft_consensus.cc:357] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.599134 1176 raft_consensus.cc:383] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.599269 1176 raft_consensus.cc:738] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.599315 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.599726 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.600595 1176 consensus_queue.cc:260] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.598887 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.600232 2340 consensus_queue.cc:1035] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:01.602160 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 05df026727da4ea594810bb9b516dc7c (DEFAULT_TABLE table=default.Table_99e81a65fa2741ac80eccc081148192a [id=ddd3c972257b48878e250cf287622605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.602660 1157 log.cc:1198] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/8890a6a93e3d48dc96955f1b51e78603
I20250109 20:53:01.602835 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05df026727da4ea594810bb9b516dc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.603083 1157 ts_tablet_manager.cc:1939] T 8890a6a93e3d48dc96955f1b51e78603 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.604702 1176 ts_tablet_manager.cc:1428] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:53:01.605242 1176 tablet_bootstrap.cc:492] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:01.605952 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 8890a6a93e3d48dc96955f1b51e78603 (table default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]) successfully deleted
W20250109 20:53:01.606690 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet fb3e620799a943c6850e20a3d6d2f6bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb3e620799a943c6850e20a3d6d2f6bb already in progress: creating tablet
W20250109 20:53:01.608042 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d4b939cdea1c4719b9b3918aabbdfc0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b939cdea1c4719b9b3918aabbdfc0e already in progress: creating tablet
W20250109 20:53:01.608712 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 67be9533983443cbb8fc52031216a973 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67be9533983443cbb8fc52031216a973 already in progress: creating tablet
I20250109 20:53:01.608790 1160 ts_tablet_manager.cc:1918] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.608914 1160 log.cc:1198] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4d2ad301c11943b69b59be18d6775513
I20250109 20:53:01.609221 1160 ts_tablet_manager.cc:1939] T 4d2ad301c11943b69b59be18d6775513 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:53:01.609715 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 062f1d04adc14599ace583896a3117e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 062f1d04adc14599ace583896a3117e0 already in progress: creating tablet
I20250109 20:53:01.609819 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4d2ad301c11943b69b59be18d6775513 (table default.Table_4f8710d177654a9c81c383e7aac0841b [id=3a1cd7e71eab4f90ae054b885cee1626]) successfully deleted
I20250109 20:53:01.609958 1160 tablet_replica.cc:331] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:01.610116 1160 raft_consensus.cc:2238] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.610266 1160 raft_consensus.cc:2267] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:01.610294 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7a0bfb44367a4572a9507af0448af846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a0bfb44367a4572a9507af0448af846 already in progress: creating tablet
W20250109 20:53:01.610359 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d4b939cdea1c4719b9b3918aabbdfc0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b939cdea1c4719b9b3918aabbdfc0e already in progress: creating tablet
W20250109 20:53:01.610575 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9f4e74a5b52948adb470c7bbc4616b49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f4e74a5b52948adb470c7bbc4616b49 already in progress: creating tablet
I20250109 20:53:01.611263 1165 tablet_bootstrap.cc:492] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:01.611375 1165 ts_tablet_manager.cc:1397] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.001s
W20250109 20:53:01.611980 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9f4e74a5b52948adb470c7bbc4616b49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f4e74a5b52948adb470c7bbc4616b49 already in progress: creating tablet
I20250109 20:53:01.611999 1165 raft_consensus.cc:357] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.612180 1165 raft_consensus.cc:383] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.612233 1165 raft_consensus.cc:738] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.612401 1157 tablet_replica.cc:331] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.612416 1165 consensus_queue.cc:260] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.612869 1157 raft_consensus.cc:2238] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:01.612903 1165 ts_tablet_manager.cc:1428] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:53:01.613657 1157 raft_consensus.cc:2267] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.614898 1165 tablet_bootstrap.cc:492] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:01.616015 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cede2a1882946a181b18d66f12c5114 with delete_type TABLET_DATA_DELETED (replaced by tablet 1343e8e822684fea8a13d8df9ec33779) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.616250 1160 ts_tablet_manager.cc:1905] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.616844 1157 ts_tablet_manager.cc:1905] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.618775 888 tablet_service.cc:1467] Processing CreateTablet for tablet bc8fa525cebd45af9b370df8a39f09b5 (DEFAULT_TABLE table=default.Table_392b71f0c7604df7acc42d6729b4d78a [id=4cf49ba7d7494fe791a42bf4b403a2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.621626 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8fa525cebd45af9b370df8a39f09b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.623574 747 tablet_service.cc:1467] Processing CreateTablet for tablet bc8fa525cebd45af9b370df8a39f09b5 (DEFAULT_TABLE table=default.Table_392b71f0c7604df7acc42d6729b4d78a [id=4cf49ba7d7494fe791a42bf4b403a2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.623890 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8fa525cebd45af9b370df8a39f09b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.624034 882 tablet_service.cc:1467] Processing CreateTablet for tablet caf806685f914510b53c076f1cfd6bc5 (DEFAULT_TABLE table=default.Table_d9871f7468704e5982b38ab253e14fef [id=419f54037c1c4f7dacbbf71a87db0144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.624289 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf806685f914510b53c076f1cfd6bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.628901 1165 tablet_bootstrap.cc:654] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.630414 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 71ac8d15d48c49289712ce27768dc27b with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.631160 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4f1499e571354faf9952af5288b5c3fa
I20250109 20:53:01.631860 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1499e571354faf9952af5288b5c3fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.631968 1032 tablet_service.cc:1467] Processing CreateTablet for tablet bc8fa525cebd45af9b370df8a39f09b5 (DEFAULT_TABLE table=default.Table_392b71f0c7604df7acc42d6729b4d78a [id=4cf49ba7d7494fe791a42bf4b403a2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.632095 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1499e571354faf9952af5288b5c3fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.632362 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1499e571354faf9952af5288b5c3fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.632436 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8fa525cebd45af9b370df8a39f09b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.635288 1157 ts_tablet_manager.cc:1918] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:01.637306 2032 raft_consensus.cc:491] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.637421 2032 raft_consensus.cc:513] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.638003 2032 leader_election.cc:290] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.638636 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 5b2fcddba2ac4d82b2d25bfc64a5967e from {username='slave'} at 127.0.0.1:33768
W20250109 20:53:01.639182 854 leader_election.cc:343] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 47726b58d6714cd1b46848a16a34ff37
W20250109 20:53:01.640288 854 leader_election.cc:343] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 47726b58d6714cd1b46848a16a34ff37
I20250109 20:53:01.640375 854 leader_election.cc:304] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.640864 2032 raft_consensus.cc:2743] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.643829 2340 raft_consensus.cc:491] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.643954 2340 raft_consensus.cc:513] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.644511 2340 leader_election.cc:290] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.644869 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9a0bcc8bdce941d5a0df1f6d57dd2050
W20250109 20:53:01.645377 713 leader_election.cc:343] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 93d409409813420689134773c627833b
I20250109 20:53:01.645470 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.646209 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.646488 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d409409813420689134773c627833b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:01.646919 712 leader_election.cc:343] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.646999 712 leader_election.cc:304] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.647352 2340 raft_consensus.cc:2743] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.647585 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.647763 1157 log.cc:1198] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/62caa775ef1241f68c0323a2d228c8cd
I20250109 20:53:01.648137 1157 ts_tablet_manager.cc:1939] T 62caa775ef1241f68c0323a2d228c8cd P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.649225 1176 tablet_bootstrap.cc:654] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.649283 1160 ts_tablet_manager.cc:1918] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:01.649379 1160 log.cc:1198] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/62caa775ef1241f68c0323a2d228c8cd
I20250109 20:53:01.649964 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]) successfully deleted
I20250109 20:53:01.650278 1157 tablet_replica.cc:331] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.650487 1157 raft_consensus.cc:2238] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.650678 1157 raft_consensus.cc:2267] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.653393 1157 ts_tablet_manager.cc:1905] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.659402 888 tablet_service.cc:1467] Processing CreateTablet for tablet e701f630d2e3488a85cce06235b641fb (DEFAULT_TABLE table=default.Table_3a0f3311269245aa8308597dae6b0428 [id=d762c21a45f240c3af95257fed5a6dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.659482 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6492caf3355c4685a6f0670ad1a352e9 (DEFAULT_TABLE table=default.Table_8c8b587893cb4890887373eb89950b96 [id=1aa0b264f207481d8c21653dfbbbf547]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.659735 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e701f630d2e3488a85cce06235b641fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.659909 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6492caf3355c4685a6f0670ad1a352e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.660354 747 tablet_service.cc:1467] Processing CreateTablet for tablet caf806685f914510b53c076f1cfd6bc5 (DEFAULT_TABLE table=default.Table_d9871f7468704e5982b38ab253e14fef [id=419f54037c1c4f7dacbbf71a87db0144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.664866 740 tablet_service.cc:1467] Processing CreateTablet for tablet 6492caf3355c4685a6f0670ad1a352e9 (DEFAULT_TABLE table=default.Table_8c8b587893cb4890887373eb89950b96 [id=1aa0b264f207481d8c21653dfbbbf547]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.665794 1026 tablet_service.cc:1467] Processing CreateTablet for tablet caf806685f914510b53c076f1cfd6bc5 (DEFAULT_TABLE table=default.Table_d9871f7468704e5982b38ab253e14fef [id=419f54037c1c4f7dacbbf71a87db0144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.666185 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf806685f914510b53c076f1cfd6bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.666631 1160 ts_tablet_manager.cc:1939] T 62caa775ef1241f68c0323a2d228c8cd P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:01.666628 756 tablet_service.cc:1467] Processing CreateTablet for tablet e701f630d2e3488a85cce06235b641fb (DEFAULT_TABLE table=default.Table_3a0f3311269245aa8308597dae6b0428 [id=d762c21a45f240c3af95257fed5a6dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.667440 1160 tablet_replica.cc:331] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:01.667420 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]) successfully deleted
I20250109 20:53:01.667600 1160 raft_consensus.cc:2238] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.667765 1160 raft_consensus.cc:2267] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:01.667930 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 67be9533983443cbb8fc52031216a973 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67be9533983443cbb8fc52031216a973 already in progress: creating tablet
W20250109 20:53:01.668177 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7a0bfb44367a4572a9507af0448af846 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a0bfb44367a4572a9507af0448af846 already in progress: creating tablet
W20250109 20:53:01.668421 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 062f1d04adc14599ace583896a3117e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 062f1d04adc14599ace583896a3117e0 already in progress: creating tablet
I20250109 20:53:01.668962 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 6492caf3355c4685a6f0670ad1a352e9 (DEFAULT_TABLE table=default.Table_8c8b587893cb4890887373eb89950b96 [id=1aa0b264f207481d8c21653dfbbbf547]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.669268 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6492caf3355c4685a6f0670ad1a352e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.671375 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b3c430aa71444401a3fee0a2e1c39582" } new_table_name: "default.Table_deeafd87e3e544a4ac810370ae55045e" modify_external_catalogs: true
I20250109 20:53:01.672951 451 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]). Expected version 3 got 0
I20250109 20:53:01.673027 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e701f630d2e3488a85cce06235b641fb (DEFAULT_TABLE table=default.Table_3a0f3311269245aa8308597dae6b0428 [id=d762c21a45f240c3af95257fed5a6dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.673321 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e701f630d2e3488a85cce06235b641fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.673899 1160 ts_tablet_manager.cc:1905] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.673988 2339 tablet.cc:1721] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:01.676788 1176 tablet_bootstrap.cc:492] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.678242 1176 ts_tablet_manager.cc:1397] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20250109 20:53:01.682117 1176 raft_consensus.cc:357] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.682355 1176 raft_consensus.cc:383] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.682430 1176 raft_consensus.cc:738] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.683873 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]). Expected version 3 got 0
I20250109 20:53:01.685729 1176 consensus_queue.cc:260] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.686987 1158 ts_tablet_manager.cc:1397] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.135s user 0.015s sys 0.006s
I20250109 20:53:01.687575 1158 raft_consensus.cc:357] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.687736 1158 raft_consensus.cc:383] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.687788 1158 raft_consensus.cc:738] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:01.690485 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 consensus state has no leader
I20250109 20:53:01.690541 1176 ts_tablet_manager.cc:1428] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250109 20:53:01.692377 1176 tablet_bootstrap.cc:492] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:01.695169 882 tablet_service.cc:1467] Processing CreateTablet for tablet f10ef1a9c7464aba86750f00bcbcb36d (DEFAULT_TABLE table=default.Table_770bfdf999cd4d70b649d37f9f16197e [id=e7336b9e2cf24505bdf7144fece33df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.695740 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10ef1a9c7464aba86750f00bcbcb36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.697687 1158 consensus_queue.cc:260] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.698676 1158 ts_tablet_manager.cc:1428] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20250109 20:53:01.698909 1166 ts_tablet_manager.cc:1918] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:01.699364 1160 ts_tablet_manager.cc:1918] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.699452 1160 log.cc:1198] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/47726b58d6714cd1b46848a16a34ff37
I20250109 20:53:01.699725 1160 ts_tablet_manager.cc:1939] T 47726b58d6714cd1b46848a16a34ff37 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:01.699980 1166 log.cc:1198] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/62caa775ef1241f68c0323a2d228c8cd
I20250109 20:53:01.700241 1158 tablet_bootstrap.cc:492] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:01.703011 742 tablet_service.cc:1514] Processing DeleteTablet for tablet cd23012af5574f37944f9e26c07d05d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.703544 1166 ts_tablet_manager.cc:1939] T 62caa775ef1241f68c0323a2d228c8cd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:01.705499 1158 tablet_bootstrap.cc:654] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.707322 1176 tablet_bootstrap.cc:654] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:01.708930 2223 log.cc:926] Time spent T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250109 20:53:01.710290 1158 tablet_bootstrap.cc:492] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:01.710901 1158 ts_tablet_manager.cc:1397] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250109 20:53:01.711261 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 62caa775ef1241f68c0323a2d228c8cd (table default.Table_142271ba6e494ddd876859927b650918 [id=9675dd7ce663428483bab06a077d9c0f]) successfully deleted
I20250109 20:53:01.713325 888 tablet_service.cc:1467] Processing CreateTablet for tablet eab1a10a66a24f16b6cff62379808d8c (DEFAULT_TABLE table=default.Table_97cc80276bc548db859d3adb65d9bfa9 [id=7ef40f1671994ee88d940e2d2fdd52aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.713657 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab1a10a66a24f16b6cff62379808d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.714555 887 tablet_service.cc:1467] Processing CreateTablet for tablet 6e895173440e48faafe5e9bdc37051b5 (DEFAULT_TABLE table=default.Table_e85779bce1cf4ab4926e6c7af5b7fa33 [id=2c512da2ff3d488cafd90ec15884d094]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.714831 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e895173440e48faafe5e9bdc37051b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.717307 881 tablet_service.cc:1467] Processing CreateTablet for tablet e15c4d4aac9445bc9e85fc7693ebdec8 (DEFAULT_TABLE table=default.Table_591218a0ea174cecb8e1b74b3ee303f5 [id=ed4f299ff3644cf2aaa9c3867b1835ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.717339 1019 tablet_service.cc:1467] Processing CreateTablet for tablet e15c4d4aac9445bc9e85fc7693ebdec8 (DEFAULT_TABLE table=default.Table_591218a0ea174cecb8e1b74b3ee303f5 [id=ed4f299ff3644cf2aaa9c3867b1835ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.717573 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15c4d4aac9445bc9e85fc7693ebdec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.717526 883 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9912ba97e04666bba5911cd3d5c296 (DEFAULT_TABLE table=default.Table_62c8e60a5c084b149750c3b8acb15aa8 [id=5878c3fe47ec409a931fd0bd121c1dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.717753 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a9912ba97e04666bba5911cd3d5c296. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.722214 1030 tablet_service.cc:1467] Processing CreateTablet for tablet eab1a10a66a24f16b6cff62379808d8c (DEFAULT_TABLE table=default.Table_97cc80276bc548db859d3adb65d9bfa9 [id=7ef40f1671994ee88d940e2d2fdd52aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.722525 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6e895173440e48faafe5e9bdc37051b5 (DEFAULT_TABLE table=default.Table_e85779bce1cf4ab4926e6c7af5b7fa33 [id=2c512da2ff3d488cafd90ec15884d094]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.722610 1032 tablet_service.cc:1467] Processing CreateTablet for tablet f10ef1a9c7464aba86750f00bcbcb36d (DEFAULT_TABLE table=default.Table_770bfdf999cd4d70b649d37f9f16197e [id=e7336b9e2cf24505bdf7144fece33df2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:01.724942 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 062f1d04adc14599ace583896a3117e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 062f1d04adc14599ace583896a3117e0 already in progress: creating tablet
I20250109 20:53:01.725055 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab1a10a66a24f16b6cff62379808d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.725468 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e895173440e48faafe5e9bdc37051b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.726225 1166 tablet_replica.cc:331] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:01.727593 1166 raft_consensus.cc:2238] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.728467 1166 raft_consensus.cc:2267] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.729105 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10ef1a9c7464aba86750f00bcbcb36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.729657 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15c4d4aac9445bc9e85fc7693ebdec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.733136 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1a2eba2c16714ea1951f0037668ef8f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.733690 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a2eba2c16714ea1951f0037668ef8f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.713505 742 tablet_service.cc:1467] Processing CreateTablet for tablet f10ef1a9c7464aba86750f00bcbcb36d (DEFAULT_TABLE table=default.Table_770bfdf999cd4d70b649d37f9f16197e [id=e7336b9e2cf24505bdf7144fece33df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.714999 745 tablet_service.cc:1467] Processing CreateTablet for tablet 6e895173440e48faafe5e9bdc37051b5 (DEFAULT_TABLE table=default.Table_e85779bce1cf4ab4926e6c7af5b7fa33 [id=2c512da2ff3d488cafd90ec15884d094]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.716574 751 tablet_service.cc:1467] Processing CreateTablet for tablet e15c4d4aac9445bc9e85fc7693ebdec8 (DEFAULT_TABLE table=default.Table_591218a0ea174cecb8e1b74b3ee303f5 [id=ed4f299ff3644cf2aaa9c3867b1835ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.716943 750 tablet_service.cc:1467] Processing CreateTablet for tablet eab1a10a66a24f16b6cff62379808d8c (DEFAULT_TABLE table=default.Table_97cc80276bc548db859d3adb65d9bfa9 [id=7ef40f1671994ee88d940e2d2fdd52aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.720975 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf806685f914510b53c076f1cfd6bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.736212 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab1a10a66a24f16b6cff62379808d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.737133 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9fe0192ffee74ba7a8398e7737878987" } new_table_name: "default.Table_f74ab9f194134e6ba390983fba09d20a" modify_external_catalogs: true
I20250109 20:53:01.740836 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15c4d4aac9445bc9e85fc7693ebdec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.741281 1165 tablet_bootstrap.cc:492] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:01.741407 1165 ts_tablet_manager.cc:1397] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.127s user 0.006s sys 0.008s
I20250109 20:53:01.741570 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6492caf3355c4685a6f0670ad1a352e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.742095 1165 raft_consensus.cc:357] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.742262 1165 raft_consensus.cc:383] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.742305 1165 raft_consensus.cc:738] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.742475 1165 consensus_queue.cc:260] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.742937 1165 ts_tablet_manager.cc:1428] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:01.745431 1165 tablet_bootstrap.cc:492] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:01.745419 2254 raft_consensus.cc:491] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.745530 2254 raft_consensus.cc:513] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.745623 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e701f630d2e3488a85cce06235b641fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.746176 2254 leader_election.cc:290] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.747046 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e46f4312844b6eb29d6718f4800f98" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:01.747411 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e895173440e48faafe5e9bdc37051b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:01.748039 713 leader_election.cc:343] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 04e46f4312844b6eb29d6718f4800f98
I20250109 20:53:01.751060 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_df479c8972204ae1a07040b5e2dea99c" } new_table_name: "default.Table_63f5bc8b110a4e77b65f5074015ab1cd" modify_external_catalogs: true
W20250109 20:53:01.752638 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0860554cac64b8882e39fa62462b1aa consensus state has no leader
W20250109 20:53:01.752785 712 leader_election.cc:343] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.752887 712 leader_election.cc:304] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.753291 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10ef1a9c7464aba86750f00bcbcb36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.757220 2254 raft_consensus.cc:2743] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.758543 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9912ba97e04666bba5911cd3d5c296 (DEFAULT_TABLE table=default.Table_62c8e60a5c084b149750c3b8acb15aa8 [id=5878c3fe47ec409a931fd0bd121c1dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.758567 745 tablet_service.cc:1467] Processing CreateTablet for tablet 35adbf5c1bfc474a88002f1fc14d96d3 (DEFAULT_TABLE table=default.Table_97f699f90bdf400b9f6c3ea5616fff83 [id=dd998383700d4db6ab8b13656e15928b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.758867 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35adbf5c1bfc474a88002f1fc14d96d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.761715 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9912ba97e04666bba5911cd3d5c296 (DEFAULT_TABLE table=default.Table_62c8e60a5c084b149750c3b8acb15aa8 [id=5878c3fe47ec409a931fd0bd121c1dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.762084 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a9912ba97e04666bba5911cd3d5c296. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.764247 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7bb113fc7cc443749bc8851afc2fc51e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.764847 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bb113fc7cc443749bc8851afc2fc51e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.767323 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f60d07bbecfd4e478a06f79f46ba125a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.767747 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f60d07bbecfd4e478a06f79f46ba125a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.770220 1158 raft_consensus.cc:357] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.770363 1166 ts_tablet_manager.cc:1905] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.770383 1158 raft_consensus.cc:383] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.770426 1158 raft_consensus.cc:738] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.772097 1158 consensus_queue.cc:260] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.747151 1165 tablet_bootstrap.cc:654] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.774467 1158 ts_tablet_manager.cc:1428] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250109 20:53:01.774675 1158 tablet_bootstrap.cc:492] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:01.775156 453 master_service.cc:946] ReplaceTablet: received request to replace tablet ceb2a882918046af95a884b87b0d084f from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:01.777017 882 tablet_service.cc:1467] Processing CreateTablet for tablet 35adbf5c1bfc474a88002f1fc14d96d3 (DEFAULT_TABLE table=default.Table_97f699f90bdf400b9f6c3ea5616fff83 [id=dd998383700d4db6ab8b13656e15928b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.779295 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7aba70359c5c46a9983c0d6750e95591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.779785 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aba70359c5c46a9983c0d6750e95591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.779367 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_809a4f599c56477ca32143d77d2788fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.780738 1157 ts_tablet_manager.cc:1918] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.780846 1157 log.cc:1198] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/57e56f30e6f3450a8426cd40db89e009
W20250109 20:53:01.780969 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_809a4f599c56477ca32143d77d2788fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.782045 747 tablet_service.cc:1514] Processing DeleteTablet for tablet ca816507d0c047668a51f6e7ebce18cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce0a941f0c14501a947a5c7e2a844e8) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.783164 1157 ts_tablet_manager.cc:1939] T 57e56f30e6f3450a8426cd40db89e009 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.783691 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 57e56f30e6f3450a8426cd40db89e009 (table default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]) successfully deleted
I20250109 20:53:01.783985 1157 tablet_replica.cc:331] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.784272 1157 raft_consensus.cc:2238] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.784443 1157 raft_consensus.cc:2267] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:01.784871 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f9145358b66d489788ec5cb9d4cd68e5 consensus state has no leader
I20250109 20:53:01.785153 1157 ts_tablet_manager.cc:1905] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.785936 1160 tablet_replica.cc:331] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:01.786036 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 47726b58d6714cd1b46848a16a34ff37 (table default.Table_f33bf5f9431146a5b111b0448a62e85d [id=3123de0f3f7644739ee589a4c7e9edec]) successfully deleted
I20250109 20:53:01.786336 1160 raft_consensus.cc:2238] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.786796 1160 raft_consensus.cc:2267] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.789054 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1d320aee9b99411eb394d0d28de07ce4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:01.789448 1176 tablet_bootstrap.cc:492] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.789578 1176 ts_tablet_manager.cc:1397] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.098s user 0.005s sys 0.000s
W20250109 20:53:01.789680 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d320aee9b99411eb394d0d28de07ce4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.790190 1176 raft_consensus.cc:357] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.790356 1176 raft_consensus.cc:383] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.790403 1176 raft_consensus.cc:738] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.790745 1176 consensus_queue.cc:260] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.791250 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35adbf5c1bfc474a88002f1fc14d96d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.792080 1157 ts_tablet_manager.cc:1918] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.792212 1157 log.cc:1198] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/93d409409813420689134773c627833b
I20250109 20:53:01.792503 1157 ts_tablet_manager.cc:1939] T 93d409409813420689134773c627833b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.796046 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5bc556c741f74a51ac51116ee71ec9cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.796568 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bc556c741f74a51ac51116ee71ec9cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.796893 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_06b47da1327e425989c115839e9aecbe" } modify_external_catalogs: true
I20250109 20:53:01.799064 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_06b47da1327e425989c115839e9aecbe" } modify_external_catalogs: true
I20250109 20:53:01.799912 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet da7f67026b164e0faad128d34188afb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.802452 446 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]). Expected version 3 got 0
I20250109 20:53:01.802969 2339 tablet.cc:1721] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:01.803387 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7f4fb38109404eeaa5a743e859c813c6" } new_table_name: "default.Table_833b545ce26c4298871281f72871e989" modify_external_catalogs: true
I20250109 20:53:01.804488 2343 tablet.cc:1721] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:01.804610 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_24614ad1648f41d9bc4bb77eb2b170ae" } modify_external_catalogs: true
I20250109 20:53:01.804780 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_24614ad1648f41d9bc4bb77eb2b170ae" } modify_external_catalogs: true
I20250109 20:53:01.805393 2254 raft_consensus.cc:491] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.805516 2254 raft_consensus.cc:513] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.806079 2254 leader_election.cc:290] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:01.806452 1176 ts_tablet_manager.cc:1428] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250109 20:53:01.806501 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8e249bfdf7084d0d838c0c7b47e75609"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.806929 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e249bfdf7084d0d838c0c7b47e75609 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.808701 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca816507d0c047668a51f6e7ebce18cb" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:01.810781 712 leader_election.cc:343] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:01.811507 713 leader_election.cc:343] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: ca816507d0c047668a51f6e7ebce18cb
I20250109 20:53:01.811600 713 leader_election.cc:304] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:01.811973 2254 raft_consensus.cc:2743] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.813632 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 35adbf5c1bfc474a88002f1fc14d96d3 (DEFAULT_TABLE table=default.Table_97f699f90bdf400b9f6c3ea5616fff83 [id=dd998383700d4db6ab8b13656e15928b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.814658 888 tablet_service.cc:1467] Processing CreateTablet for tablet 23b0e904a87e499f812d208228729baf (DEFAULT_TABLE table=default.Table_bc2f052f8a5c4572b2f58622e6dc5f29 [id=25ac2a3af14b4675b4e96c615c4f9e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.815122 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b0e904a87e499f812d208228729baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.815263 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 23b0e904a87e499f812d208228729baf (DEFAULT_TABLE table=default.Table_bc2f052f8a5c4572b2f58622e6dc5f29 [id=25ac2a3af14b4675b4e96c615c4f9e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.818332 750 tablet_service.cc:1467] Processing CreateTablet for tablet a46882e637824d139ac073686c16e46f (DEFAULT_TABLE table=default.Table_a4b4139a715c423ea5d28aa9c087d962 [id=8e21b193ddee48c5bc398fa4b9f770b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.818715 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46882e637824d139ac073686c16e46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.819639 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 93d409409813420689134773c627833b (table default.Table_1ee95fba580446b8926cbc779fca720f [id=a4a7c86d0df2412a83f12157c967aae4]) successfully deleted
I20250109 20:53:01.819937 1157 tablet_replica.cc:331] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.820119 1157 raft_consensus.cc:2238] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.820257 1157 raft_consensus.cc:2267] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.821326 883 tablet_service.cc:1467] Processing CreateTablet for tablet a7776914c6eb4b308f72ea8011d70cea (DEFAULT_TABLE table=default.Table_20316b8f5dec42f191b3384673593c52 [id=f9e700acaa40469889c2fb0bf1c1a0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.821768 881 tablet_service.cc:1467] Processing CreateTablet for tablet a46882e637824d139ac073686c16e46f (DEFAULT_TABLE table=default.Table_a4b4139a715c423ea5d28aa9c087d962 [id=8e21b193ddee48c5bc398fa4b9f770b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.822062 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7776914c6eb4b308f72ea8011d70cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.822470 1176 tablet_bootstrap.cc:492] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:01.824115 1176 tablet_bootstrap.cc:654] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.824836 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46882e637824d139ac073686c16e46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.829360 1158 tablet_bootstrap.cc:654] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.832134 2339 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:01.833794 747 tablet_service.cc:1467] Processing CreateTablet for tablet 23b0e904a87e499f812d208228729baf (DEFAULT_TABLE table=default.Table_bc2f052f8a5c4572b2f58622e6dc5f29 [id=25ac2a3af14b4675b4e96c615c4f9e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.834208 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b0e904a87e499f812d208228729baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.836933 745 tablet_service.cc:1467] Processing CreateTablet for tablet a7776914c6eb4b308f72ea8011d70cea (DEFAULT_TABLE table=default.Table_20316b8f5dec42f191b3384673593c52 [id=f9e700acaa40469889c2fb0bf1c1a0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.837288 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7776914c6eb4b308f72ea8011d70cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.837683 2318 raft_consensus.cc:491] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:01.837795 2318 raft_consensus.cc:513] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.838035 888 tablet_service.cc:1467] Processing CreateTablet for tablet b1e6beb38a994f02a6c1d7357c2dbd63 (DEFAULT_TABLE table=default.Table_0f0e855cab8543789fddd27edca161f6 [id=50ec672430f44c4fad14259b5daa27a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.838380 2318 leader_election.cc:290] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:01.838672 1017 tablet_service.cc:1467] Processing CreateTablet for tablet b1e6beb38a994f02a6c1d7357c2dbd63 (DEFAULT_TABLE table=default.Table_0f0e855cab8543789fddd27edca161f6 [id=50ec672430f44c4fad14259b5daa27a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.838821 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7f67026b164e0faad128d34188afb5" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:01.838831 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7f67026b164e0faad128d34188afb5" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:01.839299 989 leader_election.cc:343] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.839357 750 tablet_service.cc:1467] Processing CreateTablet for tablet b1e6beb38a994f02a6c1d7357c2dbd63 (DEFAULT_TABLE table=default.Table_0f0e855cab8543789fddd27edca161f6 [id=50ec672430f44c4fad14259b5daa27a0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:01.839406 990 leader_election.cc:343] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:01.839471 990 leader_election.cc:304] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:01.839630 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e6beb38a994f02a6c1d7357c2dbd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.839854 2318 raft_consensus.cc:2743] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:01.840963 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a7776914c6eb4b308f72ea8011d70cea (DEFAULT_TABLE table=default.Table_20316b8f5dec42f191b3384673593c52 [id=f9e700acaa40469889c2fb0bf1c1a0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.842535 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a9912ba97e04666bba5911cd3d5c296. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.842972 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35adbf5c1bfc474a88002f1fc14d96d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.843338 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b0e904a87e499f812d208228729baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.844081 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e6beb38a994f02a6c1d7357c2dbd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.852479 1165 tablet_bootstrap.cc:492] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:01.852665 1165 ts_tablet_manager.cc:1397] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.107s user 0.002s sys 0.002s
I20250109 20:53:01.853431 1165 raft_consensus.cc:357] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.853657 1165 raft_consensus.cc:383] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.853734 1165 raft_consensus.cc:738] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.854094 1165 consensus_queue.cc:260] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.854725 1165 ts_tablet_manager.cc:1428] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:01.857223 2343 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:01.861032 1165 tablet_bootstrap.cc:492] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:01.861212 1176 tablet_bootstrap.cc:492] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.861372 1176 ts_tablet_manager.cc:1397] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250109 20:53:01.862005 1176 raft_consensus.cc:357] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.862259 1176 raft_consensus.cc:383] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.862358 1176 raft_consensus.cc:738] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.862510 1165 tablet_bootstrap.cc:654] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.862628 1176 consensus_queue.cc:260] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.863309 1176 ts_tablet_manager.cc:1428] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:01.863610 1176 tablet_bootstrap.cc:492] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:01.864107 1157 ts_tablet_manager.cc:1905] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.879623 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e6beb38a994f02a6c1d7357c2dbd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.879699 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7776914c6eb4b308f72ea8011d70cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.883435 1032 tablet_service.cc:1467] Processing CreateTablet for tablet a46882e637824d139ac073686c16e46f (DEFAULT_TABLE table=default.Table_a4b4139a715c423ea5d28aa9c087d962 [id=8e21b193ddee48c5bc398fa4b9f770b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.883752 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46882e637824d139ac073686c16e46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.893976 1160 ts_tablet_manager.cc:1905] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.900481 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 59ec9a54da2e43bfbb22637385c53a7a (DEFAULT_TABLE table=default.Table_2768784b71f9461bbe4081deaaae9588 [id=dcef80bb6149463abb98ff6a0616cb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.900804 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ec9a54da2e43bfbb22637385c53a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.904348 1176 tablet_bootstrap.cc:654] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.905050 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:01.905771 881 tablet_service.cc:1467] Processing CreateTablet for tablet 493fcbe4c5ad4a54ae6476a99ec39b70 (DEFAULT_TABLE table=default.Table_6c494755f29b4b4ab961151e05604fa9 [id=05d3a0cc799a48ac82fba264c02a0885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.906185 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 493fcbe4c5ad4a54ae6476a99ec39b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.906474 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 493fcbe4c5ad4a54ae6476a99ec39b70 (DEFAULT_TABLE table=default.Table_6c494755f29b4b4ab961151e05604fa9 [id=05d3a0cc799a48ac82fba264c02a0885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.906816 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 493fcbe4c5ad4a54ae6476a99ec39b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.918134 1017 tablet_service.cc:1467] Processing CreateTablet for tablet fb57395c2e684e878baf53d76802dc45 (DEFAULT_TABLE table=default.Table_8b869f8172ed4fcf815c58fc296752c6 [id=838faaf3c00c4ec582e8eeb78a7db528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.918505 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb57395c2e684e878baf53d76802dc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.919706 881 tablet_service.cc:1467] Processing CreateTablet for tablet fb57395c2e684e878baf53d76802dc45 (DEFAULT_TABLE table=default.Table_8b869f8172ed4fcf815c58fc296752c6 [id=838faaf3c00c4ec582e8eeb78a7db528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.920261 882 tablet_service.cc:1467] Processing CreateTablet for tablet 59ec9a54da2e43bfbb22637385c53a7a (DEFAULT_TABLE table=default.Table_2768784b71f9461bbe4081deaaae9588 [id=dcef80bb6149463abb98ff6a0616cb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.920578 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ec9a54da2e43bfbb22637385c53a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.922792 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb57395c2e684e878baf53d76802dc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.879392 1166 ts_tablet_manager.cc:1918] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.923749 1166 log.cc:1198] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2ed7abf23ce04a268a460d8e3e97d56b
I20250109 20:53:01.924247 1166 ts_tablet_manager.cc:1939] T 2ed7abf23ce04a268a460d8e3e97d56b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:01.925129 883 tablet_service.cc:1467] Processing CreateTablet for tablet 946c48edb09e43e2bac78a1540946d92 (DEFAULT_TABLE table=default.Table_625afa33e7894342beb6fceaae41f1b6 [id=b0c2a08c478849e19444cc53d82d41c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.925405 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946c48edb09e43e2bac78a1540946d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.924659 1176 tablet_bootstrap.cc:492] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:01.929117 1176 ts_tablet_manager.cc:1397] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.000s
I20250109 20:53:01.929836 1176 raft_consensus.cc:357] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.930064 1176 raft_consensus.cc:383] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.930145 1176 raft_consensus.cc:738] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.930370 1176 consensus_queue.cc:260] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.932893 1176 ts_tablet_manager.cc:1428] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:53:01.933172 1176 tablet_bootstrap.cc:492] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:01.934638 1176 tablet_bootstrap.cc:654] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.936179 742 tablet_service.cc:1467] Processing CreateTablet for tablet 493fcbe4c5ad4a54ae6476a99ec39b70 (DEFAULT_TABLE table=default.Table_6c494755f29b4b4ab961151e05604fa9 [id=05d3a0cc799a48ac82fba264c02a0885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.936547 751 tablet_service.cc:1467] Processing CreateTablet for tablet fb57395c2e684e878baf53d76802dc45 (DEFAULT_TABLE table=default.Table_8b869f8172ed4fcf815c58fc296752c6 [id=838faaf3c00c4ec582e8eeb78a7db528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.936931 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb57395c2e684e878baf53d76802dc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.937283 747 tablet_service.cc:1467] Processing CreateTablet for tablet 59ec9a54da2e43bfbb22637385c53a7a (DEFAULT_TABLE table=default.Table_2768784b71f9461bbe4081deaaae9588 [id=dcef80bb6149463abb98ff6a0616cb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.937544 1157 ts_tablet_manager.cc:1918] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.937690 1157 log.cc:1198] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/7da1f3cafffe4f7ca39624b43831689e
I20250109 20:53:01.938030 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ec9a54da2e43bfbb22637385c53a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.938457 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 493fcbe4c5ad4a54ae6476a99ec39b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.942195 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ceb2a882918046af95a884b87b0d084f
I20250109 20:53:01.942687 453 catalog_manager.cc:6596] ReplaceTablet: tablet ceb2a882918046af95a884b87b0d084f deleted and replaced by tablet 009b90274d6b4830bf7402467f1672df
I20250109 20:53:01.944357 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_02714fd933d14554904584c58daa3159" } new_table_name: "default.Table_5789d972d13d413d984392ac1d029f75" modify_external_catalogs: true
I20250109 20:53:01.944475 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2ff361cf1b484434b63f79b7ae7da8a2" } modify_external_catalogs: true
I20250109 20:53:01.945358 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 82209a6b39bb48e791ff63a8167c9765
I20250109 20:53:01.945223 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0861a5b375f1434abfc4c91f0ae5dd2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.945745 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0861a5b375f1434abfc4c91f0ae5dd2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.945935 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d9ef5e30a4624d08b31b37a6c230d437" } modify_external_catalogs: true
I20250109 20:53:01.946122 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d9ef5e30a4624d08b31b37a6c230d437" } modify_external_catalogs: true
I20250109 20:53:01.947114 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 733c7059c750463aa586d9b733f1d1e9
I20250109 20:53:01.948791 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_487b0b221267468ba9603345f5030c8b" } modify_external_catalogs: true
I20250109 20:53:01.948938 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_487b0b221267468ba9603345f5030c8b" } modify_external_catalogs: true
I20250109 20:53:01.949406 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 82209a6b39bb48e791ff63a8167c9765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.949604 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 733c7059c750463aa586d9b733f1d1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.949733 756 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb2a882918046af95a884b87b0d084f with delete_type TABLET_DATA_DELETED (replaced by tablet 009b90274d6b4830bf7402467f1672df) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:01.949934 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b] alter complete (version 3)
I20250109 20:53:01.950253 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b67ae256dec74f84a6a39891b8e2ff60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.950651 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b67ae256dec74f84a6a39891b8e2ff60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.944654 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2ff361cf1b484434b63f79b7ae7da8a2" } modify_external_catalogs: true
W20250109 20:53:01.951651 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8c0f5dc3c794811b55040dcc094246c consensus state has no leader
I20250109 20:53:01.951932 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5b016d79eedf43bbbcdc6d742dde478e" } new_table_name: "default.Table_db486b062a6a493397542a98a8deafa8" modify_external_catalogs: true
I20250109 20:53:01.952571 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 82209a6b39bb48e791ff63a8167c9765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.952808 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 733c7059c750463aa586d9b733f1d1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.952975 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb2a882918046af95a884b87b0d084f with delete_type TABLET_DATA_DELETED (replaced by tablet 009b90274d6b4830bf7402467f1672df) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:01.953595 1158 tablet_bootstrap.cc:492] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:01.953720 450 master_service.cc:946] ReplaceTablet: received request to replace tablet f8eba3dcab764a98857604b63bf9dce1 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:01.953819 1158 ts_tablet_manager.cc:1397] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.179s user 0.004s sys 0.000s
I20250109 20:53:01.954114 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 0a2efcc2a6ec4b109845d8087491ca9f from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:01.954555 1158 raft_consensus.cc:357] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.954793 1158 raft_consensus.cc:383] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.954900 1158 raft_consensus.cc:738] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.955185 1158 consensus_queue.cc:260] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.955978 1158 ts_tablet_manager.cc:1428] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:01.956147 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 82209a6b39bb48e791ff63a8167c9765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.956331 1158 tablet_bootstrap.cc:492] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:01.956147 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 733c7059c750463aa586d9b733f1d1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.957844 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a5a6dbb823e4bd3afeb157b3c5771ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.959008 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f270a526059d4308925d102e914882ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:01.959443 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f270a526059d4308925d102e914882ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:01.960489 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_44be3f0582b34794be54b0e89682b0f1" } new_table_name: "default.Table_463c5bd1d860495f81699d4a3a47a4c5" modify_external_catalogs: true
I20250109 20:53:01.961107 1157 ts_tablet_manager.cc:1939] T 7da1f3cafffe4f7ca39624b43831689e P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:01.962101 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 7da1f3cafffe4f7ca39624b43831689e (table default.Table_4f4640c9222649fc957636279628a391 [id=cc1214f3eaf54bf0ba8167dbd8a4e634]) successfully deleted
I20250109 20:53:01.962188 1166 tablet_replica.cc:331] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:01.962325 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2ed7abf23ce04a268a460d8e3e97d56b (table default.Table_422152ed650445aeb8f7af679f9c1d12 [id=b296e8173f88419a8a14ac395611b4bc]) successfully deleted
I20250109 20:53:01.962536 1166 raft_consensus.cc:2238] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.962567 1157 tablet_replica.cc:331] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:01.962720 1157 raft_consensus.cc:2238] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:01.962836 1157 raft_consensus.cc:2267] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:01.963317 751 tablet_service.cc:1467] Processing CreateTablet for tablet 946c48edb09e43e2bac78a1540946d92 (DEFAULT_TABLE table=default.Table_625afa33e7894342beb6fceaae41f1b6 [id=b0c2a08c478849e19444cc53d82d41c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.963660 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946c48edb09e43e2bac78a1540946d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.965150 1157 ts_tablet_manager.cc:1905] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:01.972780 888 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb2a882918046af95a884b87b0d084f with delete_type TABLET_DATA_DELETED (replaced by tablet 009b90274d6b4830bf7402467f1672df) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:01.973974 1165 tablet_bootstrap.cc:492] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:01.974160 1165 ts_tablet_manager.cc:1397] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.113s user 0.001s sys 0.003s
I20250109 20:53:01.974915 1165 raft_consensus.cc:357] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.975198 1165 raft_consensus.cc:383] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:01.975292 1165 raft_consensus.cc:738] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:01.975553 1165 consensus_queue.cc:260] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:01.976109 1165 ts_tablet_manager.cc:1428] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:01.977164 1165 tablet_bootstrap.cc:492] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:01.982380 1158 tablet_bootstrap.cc:654] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:01.992539 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f8232ec65c5f499896e68d118b3c4d79
W20250109 20:53:01.992957 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bbf9224d9674b08b64f9f4e77e040e0 consensus state has no leader
I20250109 20:53:01.994530 747 tablet_service.cc:1467] Processing CreateTablet for tablet c3645f72408645ccb696a362bbf578a0 (DEFAULT_TABLE table=default.Table_62601ef08e864a52b863ba7c0d6230ac [id=8e016bc6e19f445a94d3eca1cef735b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:01.994874 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3645f72408645ccb696a362bbf578a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.996594 1157 ts_tablet_manager.cc:1918] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:01.997648 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d7b0c3e0c004488b982f368831954313" } new_table_name: "default.Table_d2938bca74b84f31a863e8a06dd3fd6b" modify_external_catalogs: true
I20250109 20:53:01.997090 2340 raft_consensus.cc:491] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.000043 2340 raft_consensus.cc:513] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.000604 2340 leader_election.cc:290] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:02.001384 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d09daa2eed41c3952fa21aa2cdd09e" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:02.001470 713 leader_election.cc:343] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: a0d09daa2eed41c3952fa21aa2cdd09e
I20250109 20:53:02.001749 1157 log.cc:1198] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/04e46f4312844b6eb29d6718f4800f98
I20250109 20:53:02.002113 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f502d666aa043e2a0b67a4e21bcf03e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:02.001957 712 leader_election.cc:343] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.004948 712 leader_election.cc:304] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:02.005388 2340 raft_consensus.cc:2743] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.002066 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f8232ec65c5f499896e68d118b3c4d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.005986 1157 ts_tablet_manager.cc:1939] T 04e46f4312844b6eb29d6718f4800f98 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:02.007097 915 raft_consensus.cc:3054] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:02.009534 1157 tablet_replica.cc:331] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:02.009847 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 04e46f4312844b6eb29d6718f4800f98 (table default.Table_c4b9fe08755e4cb1b56978cbda1f2025 [id=4394b742727948f99337b3aa45e81de4]) successfully deleted
W20250109 20:53:02.010133 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8d8220072e9f42ef90eda39e2382d2df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d8220072e9f42ef90eda39e2382d2df already in progress: creating tablet
I20250109 20:53:02.010460 1157 raft_consensus.cc:2238] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:02.011283 1157 pending_rounds.cc:62] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Trying to abort 1 pending ops.
I20250109 20:53:02.011327 1157 pending_rounds.cc:69] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523699970293760 op_type: NO_OP noop_request { }
I20250109 20:53:02.011638 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:02.013330 2339 tablet.cc:1721] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:02.018185 888 tablet_service.cc:1514] Processing DeleteTablet for tablet f8232ec65c5f499896e68d118b3c4d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.014220 1157 raft_consensus.cc:2883] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:53:02.022493 1157 raft_consensus.cc:2267] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.023226 1158 tablet_bootstrap.cc:492] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.023331 1158 ts_tablet_manager.cc:1397] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.002s
I20250109 20:53:02.023880 1158 raft_consensus.cc:357] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.024065 1158 raft_consensus.cc:383] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.024109 1158 raft_consensus.cc:738] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.024268 1158 consensus_queue.cc:260] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.024492 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0463dd5de987418d8ea1b102b09137bf
I20250109 20:53:02.024791 1158 ts_tablet_manager.cc:1428] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:02.028138 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 0463dd5de987418d8ea1b102b09137bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.028503 1158 tablet_bootstrap.cc:492] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.028848 1165 tablet_bootstrap.cc:654] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.029471 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0463dd5de987418d8ea1b102b09137bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.030273 1158 tablet_bootstrap.cc:654] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.032979 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cbf320b748bb4162a3ce1ef93588df7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.033483 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbf320b748bb4162a3ce1ef93588df7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.043421 1157 ts_tablet_manager.cc:1905] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.044426 881 tablet_service.cc:1467] Processing CreateTablet for tablet c3645f72408645ccb696a362bbf578a0 (DEFAULT_TABLE table=default.Table_62601ef08e864a52b863ba7c0d6230ac [id=8e016bc6e19f445a94d3eca1cef735b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.047061 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3645f72408645ccb696a362bbf578a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:01.993499 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 946c48edb09e43e2bac78a1540946d92 (DEFAULT_TABLE table=default.Table_625afa33e7894342beb6fceaae41f1b6 [id=b0c2a08c478849e19444cc53d82d41c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.001925 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet f8232ec65c5f499896e68d118b3c4d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.010526 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c3645f72408645ccb696a362bbf578a0 (DEFAULT_TABLE table=default.Table_62601ef08e864a52b863ba7c0d6230ac [id=8e016bc6e19f445a94d3eca1cef735b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.052269 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3645f72408645ccb696a362bbf578a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.013818 2343 tablet.cc:1721] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:02.030251 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0463dd5de987418d8ea1b102b09137bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.055976 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2dfbf64b178841d7aaf34632cbff3716
I20250109 20:53:02.057176 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfbf64b178841d7aaf34632cbff3716 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.057442 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfbf64b178841d7aaf34632cbff3716 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.059371 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946c48edb09e43e2bac78a1540946d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.060058 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfbf64b178841d7aaf34632cbff3716 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.048151 1176 tablet_bootstrap.cc:492] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:02.064666 1176 ts_tablet_manager.cc:1397] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.000s
I20250109 20:53:02.065613 1176 raft_consensus.cc:357] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.065868 1176 raft_consensus.cc:383] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.066022 1176 raft_consensus.cc:738] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.066277 1176 consensus_queue.cc:260] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.068467 2369 raft_consensus.cc:491] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.068750 2369 raft_consensus.cc:513] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.069593 1166 raft_consensus.cc:2267] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.070241 1166 ts_tablet_manager.cc:1905] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.072031 1157 ts_tablet_manager.cc:1918] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:53:02.072373 2369 leader_election.cc:290] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:02.072413 1157 log.cc:1198] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/daac99da5d10475c9de263bef27baeb7
I20250109 20:53:02.072903 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cede2a1882946a181b18d66f12c5114" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:02.073444 990 leader_election.cc:343] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.073704 1157 ts_tablet_manager.cc:1939] T daac99da5d10475c9de263bef27baeb7 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:02.075017 2032 raft_consensus.cc:491] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.075165 2032 raft_consensus.cc:513] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.075747 2032 leader_election.cc:290] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:02.076493 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet daac99da5d10475c9de263bef27baeb7 (table default.Table_d70feebb4775482eb8b3b251c50ad92c [id=27a0981b71084436994f1e9689afb145]) successfully deleted
W20250109 20:53:02.077607 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f77f35085ff43a5b7fb6cce7e7a3409 already in progress: creating tablet
W20250109 20:53:02.078136 854 leader_election.cc:343] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 0f502d666aa043e2a0b67a4e21bcf03e
W20250109 20:53:02.078541 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4f1499e571354faf9952af5288b5c3fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1499e571354faf9952af5288b5c3fa already in progress: creating tablet
W20250109 20:53:02.079288 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 already in progress: creating tablet
I20250109 20:53:02.079767 1157 tablet_replica.cc:331] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:02.080281 1157 raft_consensus.cc:2238] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.080628 1157 raft_consensus.cc:2267] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:02.082412 854 leader_election.cc:343] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 0f502d666aa043e2a0b67a4e21bcf03e
I20250109 20:53:02.082507 854 leader_election.cc:304] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:02.082818 2032 raft_consensus.cc:2743] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.091589 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7b2ba41100634eabbefe5b301dc507bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.092196 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b2ba41100634eabbefe5b301dc507bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.093560 1165 tablet_bootstrap.cc:492] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.093693 1165 ts_tablet_manager.cc:1397] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.117s user 0.001s sys 0.003s
I20250109 20:53:02.096246 1165 raft_consensus.cc:357] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.096446 1165 raft_consensus.cc:383] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.096491 1165 raft_consensus.cc:738] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.098405 454 master_service.cc:946] ReplaceTablet: received request to replace tablet bba303032679462c8b16f706096b6d4d from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:02.098695 1157 ts_tablet_manager.cc:1905] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:02.101687 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7357b4f67dc6448dbb8f77dce81731f3 consensus state has no leader
I20250109 20:53:02.104305 747 tablet_service.cc:1467] Processing CreateTablet for tablet 5f05e17ca65243a898a7c75d3f453706 (DEFAULT_TABLE table=default.Table_50f7e7cb701c4134ae4844b388bf458a [id=7bc87e515a2e43e195160912c658c054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.104676 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f05e17ca65243a898a7c75d3f453706. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.105455 893 tablet_service.cc:1467] Processing CreateTablet for tablet 204ae923b8354fc791b5c4fb7197dc76 (DEFAULT_TABLE table=default.Table_d8dc3eb6434742bd9f61b5989fa3a6c7 [id=1962295f5b9b48ea94e93f8b75203039]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.105803 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 204ae923b8354fc791b5c4fb7197dc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.107987 1158 tablet_bootstrap.cc:492] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.108103 1158 ts_tablet_manager.cc:1397] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.000s
I20250109 20:53:02.109736 885 tablet_service.cc:1467] Processing CreateTablet for tablet e62c60a0b28c4bd7b4ebe7aee2adb9f2 (DEFAULT_TABLE table=default.Table_4eca95f945a544c289d09435d20f3202 [id=2a1449809a9d46dc8c347f6772abad63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.109934 883 tablet_service.cc:1467] Processing CreateTablet for tablet dd1abcc0bba34cda917d6d42de565b0c (DEFAULT_TABLE table=default.Table_99b31f42d9d14193b7b8b9809d39c458 [id=e84b01fd821d4153892910fb2382a76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.110107 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62c60a0b28c4bd7b4ebe7aee2adb9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.110390 890 tablet_service.cc:1467] Processing CreateTablet for tablet 644330b0d60e436fa1dbcfea53beee71 (DEFAULT_TABLE table=default.Table_f45cf4921b02449da770ba575ad3043d [id=40dc5b91b09d460fa99e34e23842dd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.110694 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644330b0d60e436fa1dbcfea53beee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.112593 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd1abcc0bba34cda917d6d42de565b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.112918 895 tablet_service.cc:1467] Processing CreateTablet for tablet 407c57dfd4134eeba07f74386540f844 (DEFAULT_TABLE table=default.Table_f8eaab37b5024ccd99aa643e72cdda0a [id=ac6293f1f6fe4760ae7f07e3c925081f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.113001 1023 tablet_service.cc:1467] Processing CreateTablet for tablet dd1abcc0bba34cda917d6d42de565b0c (DEFAULT_TABLE table=default.Table_99b31f42d9d14193b7b8b9809d39c458 [id=e84b01fd821d4153892910fb2382a76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.113209 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407c57dfd4134eeba07f74386540f844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.109736 887 tablet_service.cc:1467] Processing CreateTablet for tablet ac2ff099cd964e0ea946542a4f3cf619 (DEFAULT_TABLE table=default.Table_4b14acbb79d24c09be2fe93f8f1e2c16 [id=3d01803438f641d4bb1c72a1c21bfcac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.113729 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2ff099cd964e0ea946542a4f3cf619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.117022 1020 tablet_service.cc:1467] Processing CreateTablet for tablet ac2ff099cd964e0ea946542a4f3cf619 (DEFAULT_TABLE table=default.Table_4b14acbb79d24c09be2fe93f8f1e2c16 [id=3d01803438f641d4bb1c72a1c21bfcac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.118566 750 tablet_service.cc:1467] Processing CreateTablet for tablet 204ae923b8354fc791b5c4fb7197dc76 (DEFAULT_TABLE table=default.Table_d8dc3eb6434742bd9f61b5989fa3a6c7 [id=1962295f5b9b48ea94e93f8b75203039]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.118894 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 204ae923b8354fc791b5c4fb7197dc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.120167 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 204ae923b8354fc791b5c4fb7197dc76 (DEFAULT_TABLE table=default.Table_d8dc3eb6434742bd9f61b5989fa3a6c7 [id=1962295f5b9b48ea94e93f8b75203039]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.121771 751 tablet_service.cc:1467] Processing CreateTablet for tablet dd1abcc0bba34cda917d6d42de565b0c (DEFAULT_TABLE table=default.Table_99b31f42d9d14193b7b8b9809d39c458 [id=e84b01fd821d4153892910fb2382a76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.122104 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd1abcc0bba34cda917d6d42de565b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.122782 745 tablet_service.cc:1467] Processing CreateTablet for tablet ac2ff099cd964e0ea946542a4f3cf619 (DEFAULT_TABLE table=default.Table_4b14acbb79d24c09be2fe93f8f1e2c16 [id=3d01803438f641d4bb1c72a1c21bfcac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.123082 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2ff099cd964e0ea946542a4f3cf619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.123885 888 tablet_service.cc:1467] Processing CreateTablet for tablet 5f05e17ca65243a898a7c75d3f453706 (DEFAULT_TABLE table=default.Table_50f7e7cb701c4134ae4844b388bf458a [id=7bc87e515a2e43e195160912c658c054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.124154 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f05e17ca65243a898a7c75d3f453706. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.128122 742 tablet_service.cc:1467] Processing CreateTablet for tablet 644330b0d60e436fa1dbcfea53beee71 (DEFAULT_TABLE table=default.Table_f45cf4921b02449da770ba575ad3043d [id=40dc5b91b09d460fa99e34e23842dd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.128424 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644330b0d60e436fa1dbcfea53beee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.131515 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 5f05e17ca65243a898a7c75d3f453706 (DEFAULT_TABLE table=default.Table_50f7e7cb701c4134ae4844b388bf458a [id=7bc87e515a2e43e195160912c658c054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.132977 756 tablet_service.cc:1467] Processing CreateTablet for tablet 407c57dfd4134eeba07f74386540f844 (DEFAULT_TABLE table=default.Table_f8eaab37b5024ccd99aa643e72cdda0a [id=ac6293f1f6fe4760ae7f07e3c925081f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.133271 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407c57dfd4134eeba07f74386540f844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.134054 1165 consensus_queue.cc:260] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:02.136899 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 53224.5us
I20250109 20:53:02.136930 746 tablet_service.cc:1467] Processing CreateTablet for tablet e62c60a0b28c4bd7b4ebe7aee2adb9f2 (DEFAULT_TABLE table=default.Table_4eca95f945a544c289d09435d20f3202 [id=2a1449809a9d46dc8c347f6772abad63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.137223 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62c60a0b28c4bd7b4ebe7aee2adb9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.140422 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c3aa20539740cba4e5d6e852ad2909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.141810 1160 ts_tablet_manager.cc:1918] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.141924 1160 log.cc:1198] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/91dbccae0af24a629600a92e7eadfee7
I20250109 20:53:02.142216 1160 ts_tablet_manager.cc:1939] T 91dbccae0af24a629600a92e7eadfee7 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:02.109017 1158 raft_consensus.cc:357] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.142764 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 91dbccae0af24a629600a92e7eadfee7 (table default.Table_534eb61fef3c42c8bac87ac39a09ef45 [id=0e350177d27e4c8b8bc3cede2166517e]) successfully deleted
I20250109 20:53:02.142936 1158 raft_consensus.cc:383] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.143054 1158 raft_consensus.cc:738] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.143333 1158 consensus_queue.cc:260] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.144979 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0a2efcc2a6ec4b109845d8087491ca9f
I20250109 20:53:02.145428 450 catalog_manager.cc:6596] ReplaceTablet: tablet 0a2efcc2a6ec4b109845d8087491ca9f deleted and replaced by tablet 3fc68008840d4b8fae5185718ab454fa
I20250109 20:53:02.146178 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2efcc2a6ec4b109845d8087491ca9f with delete_type TABLET_DATA_DELETED (replaced by tablet 3fc68008840d4b8fae5185718ab454fa) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.146929 1158 ts_tablet_manager.cc:1428] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.039s user 0.003s sys 0.000s
I20250109 20:53:02.149997 1165 ts_tablet_manager.cc:1428] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.054s user 0.001s sys 0.000s
I20250109 20:53:02.150314 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2efcc2a6ec4b109845d8087491ca9f with delete_type TABLET_DATA_DELETED (replaced by tablet 3fc68008840d4b8fae5185718ab454fa) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.153649 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 644330b0d60e436fa1dbcfea53beee71 (DEFAULT_TABLE table=default.Table_f45cf4921b02449da770ba575ad3043d [id=40dc5b91b09d460fa99e34e23842dd54]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:02.154470 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8d8220072e9f42ef90eda39e2382d2df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d8220072e9f42ef90eda39e2382d2df already in progress: creating tablet
W20250109 20:53:02.154708 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet daac99da5d10475c9de263bef27baeb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daac99da5d10475c9de263bef27baeb7 already in progress: creating tablet
I20250109 20:53:02.155098 2340 raft_consensus.cc:491] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.155221 2340 raft_consensus.cc:513] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.155824 2340 leader_election.cc:290] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:02.156139 1157 ts_tablet_manager.cc:1918] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.156458 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ebeffe516c4c239ae2ee9040276e53" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:02.157431 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f77f35085ff43a5b7fb6cce7e7a3409 already in progress: creating tablet
I20250109 20:53:02.158334 895 tablet_service.cc:1467] Processing CreateTablet for tablet fdb0ebfad2de4fbf89a4b53a0a150906 (DEFAULT_TABLE table=default.Table_f2e044c26f7c4b9d985304d701432239 [id=ac9488e51dbf45de8c2b8ba3de3991a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.158648 1160 tablet_replica.cc:331] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:02.158792 1160 raft_consensus.cc:2238] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.159592 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e62c60a0b28c4bd7b4ebe7aee2adb9f2 (DEFAULT_TABLE table=default.Table_4eca95f945a544c289d09435d20f3202 [id=2a1449809a9d46dc8c347f6772abad63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.159868 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2efcc2a6ec4b109845d8087491ca9f with delete_type TABLET_DATA_DELETED (replaced by tablet 3fc68008840d4b8fae5185718ab454fa) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.158336 1165 tablet_bootstrap.cc:492] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:02.166544 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f83de64baea4492f9c52490b335fbd34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.167098 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f83de64baea4492f9c52490b335fbd34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.168114 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 407c57dfd4134eeba07f74386540f844 (DEFAULT_TABLE table=default.Table_f8eaab37b5024ccd99aa643e72cdda0a [id=ac6293f1f6fe4760ae7f07e3c925081f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.170894 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd1abcc0bba34cda917d6d42de565b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.171061 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1359ff2be59e4f6683ac7b4577a85027"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.171519 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1359ff2be59e4f6683ac7b4577a85027 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.171684 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2ff099cd964e0ea946542a4f3cf619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.171957 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_15d6655499224a0487b0c1f7feb8e3b9" } modify_external_catalogs: true
I20250109 20:53:02.167822 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb0ebfad2de4fbf89a4b53a0a150906. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.173549 1034 tablet_service.cc:1467] Processing CreateTablet for tablet fdb0ebfad2de4fbf89a4b53a0a150906 (DEFAULT_TABLE table=default.Table_f2e044c26f7c4b9d985304d701432239 [id=ac9488e51dbf45de8c2b8ba3de3991a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.174041 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 204ae923b8354fc791b5c4fb7197dc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.174710 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb0ebfad2de4fbf89a4b53a0a150906. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.158923 1160 raft_consensus.cc:2267] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.180508 1160 ts_tablet_manager.cc:1905] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.187853 2371 raft_consensus.cc:491] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.188297 1176 ts_tablet_manager.cc:1428] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.124s user 0.005s sys 0.000s
I20250109 20:53:02.189018 1176 tablet_bootstrap.cc:492] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:02.188067 2371 raft_consensus.cc:513] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.190407 2371 leader_election.cc:290] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:02.191488 1176 tablet_bootstrap.cc:654] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:02.157003 712 leader_election.cc:343] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.157363 742 tablet_service.cc:1467] Processing CreateTablet for tablet fdb0ebfad2de4fbf89a4b53a0a150906 (DEFAULT_TABLE table=default.Table_f2e044c26f7c4b9d985304d701432239 [id=ac9488e51dbf45de8c2b8ba3de3991a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.160063 1158 tablet_bootstrap.cc:492] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.193917 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdb0ebfad2de4fbf89a4b53a0a150906. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.194268 1158 tablet_bootstrap.cc:654] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.194861 2371 raft_consensus.cc:491] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.197317 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f05e17ca65243a898a7c75d3f453706. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.199295 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3911b54a040144baadfc4e0088d58e64" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:02.199925 990 leader_election.cc:343] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.201277 1157 log.cc:1198] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/cd23012af5574f37944f9e26c07d05d2
I20250109 20:53:02.203333 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644330b0d60e436fa1dbcfea53beee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.203739 1157 ts_tablet_manager.cc:1939] T cd23012af5574f37944f9e26c07d05d2 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:02.204576 1157 tablet_replica.cc:331] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:02.204978 1165 tablet_bootstrap.cc:654] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.206233 1157 raft_consensus.cc:2238] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.208314 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e62c60a0b28c4bd7b4ebe7aee2adb9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.208563 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407c57dfd4134eeba07f74386540f844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.209064 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet cd23012af5574f37944f9e26c07d05d2 (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:53:02.210235 1157 raft_consensus.cc:2267] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.216885 2368 raft_consensus.cc:491] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.217101 2368 raft_consensus.cc:513] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.217216 1157 ts_tablet_manager.cc:1905] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.222765 2369 raft_consensus.cc:491] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.222894 2369 raft_consensus.cc:513] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.223254 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f] alter complete (version 1)
I20250109 20:53:02.223511 2369 leader_election.cc:290] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:02.223996 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1326ea6538f74cb28d688c06eda263df" } modify_external_catalogs: true
I20250109 20:53:02.224429 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 712acab92e0f44a3a585d7d871498aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e020eeb03e041ed977779a07de5bb80) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.224640 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet e716270ffcf44039b075023fd0c2d2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.224647 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:02.225107 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a4fac4fff616461d9e0b116e98bdd5e8" } new_table_name: "default.Table_cc6cd4530e194c65892d6be31d7a1dbb" modify_external_catalogs: true
W20250109 20:53:02.225173 990 leader_election.cc:343] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.225289 742 tablet_service.cc:1514] Processing DeleteTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.218096 2371 raft_consensus.cc:513] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.227303 2371 leader_election.cc:290] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:53:02.228600 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20d27588609f465c8dedae173e57e9b2 consensus state has no leader
I20250109 20:53:02.229146 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_43b9440786c14eb0af7ca7e0aadf887b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.229769 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43b9440786c14eb0af7ca7e0aadf887b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.232318 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1326ea6538f74cb28d688c06eda263df" } modify_external_catalogs: true
W20250109 20:53:02.234052 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c785c74ecee84e6fb5d27560cc7c6f28 consensus state has no leader
I20250109 20:53:02.234402 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_90f05f2d38db4882b476617d20ba76ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.234854 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90f05f2d38db4882b476617d20ba76ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.235378 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712acab92e0f44a3a585d7d871498aa3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:02.235842 990 leader_election.cc:343] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.235885 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_15d6655499224a0487b0c1f7feb8e3b9" } modify_external_catalogs: true
I20250109 20:53:02.237296 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0ad28a15c08542d3bdf5e80b70702d85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.237715 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ad28a15c08542d3bdf5e80b70702d85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:02.243660 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 180249us
I20250109 20:53:02.244375 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:02.244884 990 leader_election.cc:343] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.245235 2368 leader_election.cc:290] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:02.247637 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cede2a1882946a181b18d66f12c5114" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:02.248088 1160 ts_tablet_manager.cc:1918] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:53:02.251173 713 leader_election.cc:343] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 59ebeffe516c4c239ae2ee9040276e53
I20250109 20:53:02.251279 713 leader_election.cc:304] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:02.251454 2340 raft_consensus.cc:2743] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:02.251499 989 leader_election.cc:343] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.251897 1160 log.cc:1198] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/71ac8d15d48c49289712ce27768dc27b
I20250109 20:53:02.251943 989 leader_election.cc:304] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:02.248059 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3911b54a040144baadfc4e0088d58e64" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:02.252245 1160 ts_tablet_manager.cc:1939] T 71ac8d15d48c49289712ce27768dc27b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:02.252377 2032 raft_consensus.cc:491] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.252492 2032 raft_consensus.cc:513] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.253038 2032 leader_election.cc:290] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:02.253597 1160 tablet_replica.cc:331] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:02.248430 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:02.253746 1160 raft_consensus.cc:2238] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.253876 1160 raft_consensus.cc:2267] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:02.253980 854 leader_election.cc:343] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 8a5a6dbb823e4bd3afeb157b3c5771ba
I20250109 20:53:02.253974 2370 raft_consensus.cc:2743] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.254487 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 71ac8d15d48c49289712ce27768dc27b (table default.Table_dc41b4efb2f245f29da9d86c1d4aef95 [id=66cee7e610614650815fbe6d06400e06]) successfully deleted
W20250109 20:53:02.254688 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4f1499e571354faf9952af5288b5c3fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1499e571354faf9952af5288b5c3fa already in progress: creating tablet
W20250109 20:53:02.254850 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 already in progress: creating tablet
W20250109 20:53:02.255025 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 82209a6b39bb48e791ff63a8167c9765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82209a6b39bb48e791ff63a8167c9765 already in progress: creating tablet
W20250109 20:53:02.255254 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 733c7059c750463aa586d9b733f1d1e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 733c7059c750463aa586d9b733f1d1e9 already in progress: creating tablet
W20250109 20:53:02.256168 989 leader_election.cc:343] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.256762 1166 ts_tablet_manager.cc:1918] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.256927 1166 log.cc:1198] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/d57a8bbb3da84088a98c816102e4e9df
I20250109 20:53:02.257328 1166 ts_tablet_manager.cc:1939] T d57a8bbb3da84088a98c816102e4e9df P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:53:02.257771 854 leader_election.cc:343] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8a5a6dbb823e4bd3afeb157b3c5771ba
I20250109 20:53:02.257846 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet d57a8bbb3da84088a98c816102e4e9df (table default.Table_634a6ecf6a7b4731ae4ac6e03c5375d1 [id=0bf46974692d47c38befe7eca4f23cc9]) successfully deleted
I20250109 20:53:02.257863 854 leader_election.cc:304] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:02.257998 854 raft_consensus.cc:2681] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:53:02.260051 1157 ts_tablet_manager.cc:1918] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.260679 1160 ts_tablet_manager.cc:1905] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.256445 989 leader_election.cc:304] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:02.265270 2370 raft_consensus.cc:2743] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:02.265581 989 leader_election.cc:343] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.265866 989 leader_election.cc:304] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:02.266175 2370 raft_consensus.cc:2743] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.248785 2374 tablet.cc:1721] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:02.248809 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712acab92e0f44a3a585d7d871498aa3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:02.250756 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:02.277667 1157 log.cc:1198] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ca816507d0c047668a51f6e7ebce18cb
W20250109 20:53:02.277902 989 leader_election.cc:343] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.278007 1157 ts_tablet_manager.cc:1939] T ca816507d0c047668a51f6e7ebce18cb P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:02.278126 989 leader_election.cc:304] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:02.278420 2370 raft_consensus.cc:2743] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:02.278856 989 leader_election.cc:343] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.279048 989 leader_election.cc:304] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:02.279192 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ca816507d0c047668a51f6e7ebce18cb (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]) successfully deleted
I20250109 20:53:02.279304 1157 tablet_replica.cc:331] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:02.279299 2370 raft_consensus.cc:2743] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:02.279417 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 82209a6b39bb48e791ff63a8167c9765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82209a6b39bb48e791ff63a8167c9765 already in progress: creating tablet
I20250109 20:53:02.279459 1157 raft_consensus.cc:2238] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.279606 1157 raft_consensus.cc:2267] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:02.279611 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 733c7059c750463aa586d9b733f1d1e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 733c7059c750463aa586d9b733f1d1e9 already in progress: creating tablet
I20250109 20:53:02.279716 1166 tablet_replica.cc:331] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:53:02.280018 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ceb2a882918046af95a884b87b0d084f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceb2a882918046af95a884b87b0d084f already in progress: creating tablet
I20250109 20:53:02.280086 1157 ts_tablet_manager.cc:1905] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:02.280242 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f8232ec65c5f499896e68d118b3c4d79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8232ec65c5f499896e68d118b3c4d79 already in progress: creating tablet
I20250109 20:53:02.277026 1158 tablet_bootstrap.cc:492] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.280576 1166 raft_consensus.cc:2238] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.280884 1158 ts_tablet_manager.cc:1397] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.121s user 0.003s sys 0.013s
I20250109 20:53:02.281596 1158 raft_consensus.cc:357] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.281839 1158 raft_consensus.cc:383] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.281867 1176 tablet_bootstrap.cc:492] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:02.281930 1158 raft_consensus.cc:738] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.282127 1176 ts_tablet_manager.cc:1397] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.000s
I20250109 20:53:02.282148 1158 consensus_queue.cc:260] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.282620 1158 ts_tablet_manager.cc:1428] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:53:02.282868 1158 tablet_bootstrap.cc:492] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.284284 1158 tablet_bootstrap.cc:654] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.286201 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:02.289170 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:02.300195 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_86f22049bc704f91b332d2205d149706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.300699 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86f22049bc704f91b332d2205d149706 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.303033 1176 raft_consensus.cc:357] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.303433 1176 raft_consensus.cc:383] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.303566 1176 raft_consensus.cc:738] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.303992 1176 consensus_queue.cc:260] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:02.309679 990 consensus_peers.cc:487] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Couldn't send request to peer 09833ea46b7141d58c3899dace69e49f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:02.281167 1166 raft_consensus.cc:2267] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.313179 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_57ac5b97c0f941e6bfa7c994e2d81482" } new_table_name: "default.Table_c8e4c2ce5b514c9e80d128f718d87c09" modify_external_catalogs: true
I20250109 20:53:02.313472 1157 ts_tablet_manager.cc:1918] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:02.313576 1157 log.cc:1198] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/0463dd5de987418d8ea1b102b09137bf
I20250109 20:53:02.313889 1157 ts_tablet_manager.cc:1939] T 0463dd5de987418d8ea1b102b09137bf P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:02.314836 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_487b0b221267468ba9603345f5030c8b [id=50837eb593734f66a03e29d8edbc63a4]) successfully deleted
I20250109 20:53:02.314999 1157 tablet_replica.cc:331] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:02.315186 1157 raft_consensus.cc:2238] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.315320 1157 raft_consensus.cc:2267] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:02.315366 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2dfbf64b178841d7aaf34632cbff3716 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dfbf64b178841d7aaf34632cbff3716 already in progress: creating tablet
W20250109 20:53:02.315570 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0a2efcc2a6ec4b109845d8087491ca9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2efcc2a6ec4b109845d8087491ca9f already in progress: creating tablet
I20250109 20:53:02.323479 1176 ts_tablet_manager.cc:1428] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20250109 20:53:02.329118 1158 tablet_bootstrap.cc:492] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.329248 1158 ts_tablet_manager.cc:1397] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20250109 20:53:02.329111 1166 ts_tablet_manager.cc:1905] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.329016 1176 tablet_bootstrap.cc:492] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:02.329941 1158 raft_consensus.cc:357] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.330116 1158 raft_consensus.cc:383] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.330186 1158 raft_consensus.cc:738] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.330369 1158 consensus_queue.cc:260] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.330962 1158 ts_tablet_manager.cc:1428] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:02.331655 1160 ts_tablet_manager.cc:1918] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.331804 1160 log.cc:1198] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/8a5a6dbb823e4bd3afeb157b3c5771ba
I20250109 20:53:02.332158 1160 ts_tablet_manager.cc:1939] T 8a5a6dbb823e4bd3afeb157b3c5771ba P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:02.332741 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 8a5a6dbb823e4bd3afeb157b3c5771ba (table default.Table_e677fcf0892746659b8f6d9f5dce6bc1 [id=d8d809ef804f4f909b825f92a823fb8d]) successfully deleted
W20250109 20:53:02.333432 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ceb2a882918046af95a884b87b0d084f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceb2a882918046af95a884b87b0d084f already in progress: creating tablet
I20250109 20:53:02.333729 1160 tablet_replica.cc:331] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:02.333925 1160 raft_consensus.cc:2238] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.334113 1160 raft_consensus.cc:2267] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.334807 1160 ts_tablet_manager.cc:1905] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.336577 1158 tablet_bootstrap.cc:492] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:02.338521 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e233caae7679488d9b61f34e5f57b4d3 (table default.Table_aef4bd2a519c4d27964b01f6449b516b [id=a231ba433af748afa4c0fca588715b59], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e233caae7679488d9b61f34e5f57b4d3 consensus state has no leader
I20250109 20:53:02.340317 2032 raft_consensus.cc:491] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.340432 2032 raft_consensus.cc:513] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.340924 2032 leader_election.cc:290] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:02.341535 1165 tablet_bootstrap.cc:492] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.341672 1165 ts_tablet_manager.cc:1397] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.183s user 0.001s sys 0.003s
I20250109 20:53:02.342381 1165 raft_consensus.cc:357] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.342572 1165 raft_consensus.cc:383] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.342628 1165 raft_consensus.cc:738] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.342820 1165 consensus_queue.cc:260] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.343284 1165 ts_tablet_manager.cc:1428] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:02.343792 1165 tablet_bootstrap.cc:492] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:02.343873 1157 ts_tablet_manager.cc:1905] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.345209 1165 tablet_bootstrap.cc:654] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:02.349625 854 leader_election.cc:343] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 06c3aa20539740cba4e5d6e852ad2909
I20250109 20:53:02.356381 1157 ts_tablet_manager.cc:1918] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.363756 1157 log.cc:1198] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/a0d09daa2eed41c3952fa21aa2cdd09e
W20250109 20:53:02.363829 854 leader_election.cc:343] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 06c3aa20539740cba4e5d6e852ad2909
I20250109 20:53:02.363976 854 leader_election.cc:304] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:02.364085 1157 ts_tablet_manager.cc:1939] T a0d09daa2eed41c3952fa21aa2cdd09e P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:02.366827 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet a0d09daa2eed41c3952fa21aa2cdd09e (table default.Table_14ad6cd05135486e87607f62e9dd62c4 [id=eccfc40a5a774588b25444c9d6f1507d]) successfully deleted
I20250109 20:53:02.368860 2032 raft_consensus.cc:2743] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.372838 1158 tablet_bootstrap.cc:654] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:02.381903 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 527375706fef48fda1adeeddf2434cb5 consensus state has no leader
I20250109 20:53:02.385841 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_86fde8a5cdef40daa1e0e95b33e93710" } modify_external_catalogs: true
I20250109 20:53:02.386051 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_86fde8a5cdef40daa1e0e95b33e93710" } modify_external_catalogs: true
I20250109 20:53:02.415022 1176 tablet_bootstrap.cc:654] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.419309 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d3428128a5494341b1986ce6956bb88a
I20250109 20:53:02.420106 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b43e8c35fcdf4973ada77b9c4ad44ba0" } modify_external_catalogs: true
I20250109 20:53:02.420342 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b43e8c35fcdf4973ada77b9c4ad44ba0" } modify_external_catalogs: true
I20250109 20:53:02.420446 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet d3428128a5494341b1986ce6956bb88a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.421578 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d3428128a5494341b1986ce6956bb88a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:02.422420 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d3428128a5494341b1986ce6956bb88a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3428128a5494341b1986ce6956bb88a already in progress: creating tablet
I20250109 20:53:02.423149 1158 tablet_bootstrap.cc:492] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.423729 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d3428128a5494341b1986ce6956bb88a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.424837 1158 ts_tablet_manager.cc:1397] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.088s user 0.000s sys 0.004s
I20250109 20:53:02.425729 1158 raft_consensus.cc:357] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.426803 1158 raft_consensus.cc:383] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.426909 1158 raft_consensus.cc:738] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.427137 1158 consensus_queue.cc:260] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.429102 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fc10e7e4961046c1b63ae98f857f8ff1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.429673 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc10e7e4961046c1b63ae98f857f8ff1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.427707 1158 ts_tablet_manager.cc:1428] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:53:02.430493 1158 tablet_bootstrap.cc:492] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.439308 1158 tablet_bootstrap.cc:654] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.442373 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ef0f27e5506444fd996d9ded42dfacb7
I20250109 20:53:02.443140 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c787d10220c41dca63172c9b3bf70ae" } modify_external_catalogs: true
I20250109 20:53:02.443318 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c787d10220c41dca63172c9b3bf70ae" } modify_external_catalogs: true
I20250109 20:53:02.444434 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0f27e5506444fd996d9ded42dfacb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.444685 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0f27e5506444fd996d9ded42dfacb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:02.445254 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ef0f27e5506444fd996d9ded42dfacb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef0f27e5506444fd996d9ded42dfacb7 already in progress: creating tablet
I20250109 20:53:02.448189 1160 ts_tablet_manager.cc:1918] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.449189 1176 tablet_bootstrap.cc:492] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:02.449370 1176 ts_tablet_manager.cc:1397] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.122s user 0.004s sys 0.000s
I20250109 20:53:02.450177 1176 raft_consensus.cc:357] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.451823 1176 raft_consensus.cc:383] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.451920 1176 raft_consensus.cc:738] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.452150 1176 consensus_queue.cc:260] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.452834 1176 ts_tablet_manager.cc:1428] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 20:53:02.462280 1176 tablet_bootstrap.cc:492] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:02.462512 2370 raft_consensus.cc:491] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.462615 2370 raft_consensus.cc:513] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.463186 2370 leader_election.cc:290] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:02.463691 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fb1017fe6c4f5abd640ad1f6214232" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:02.463872 775 raft_consensus.cc:2463] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:53:02.464011 1160 log.cc:1198] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/0f502d666aa043e2a0b67a4e21bcf03e
I20250109 20:53:02.464300 1160 ts_tablet_manager.cc:1939] T 0f502d666aa043e2a0b67a4e21bcf03e P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:02.464242 1166 ts_tablet_manager.cc:1918] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.464813 990 leader_election.cc:304] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:02.465014 1166 log.cc:1198] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/daac99da5d10475c9de263bef27baeb7
I20250109 20:53:02.465209 2370 raft_consensus.cc:2798] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:02.466003 1176 tablet_bootstrap.cc:654] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.466043 2370 raft_consensus.cc:491] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:02.466105 2370 raft_consensus.cc:3054] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:02.466272 1984 raft_consensus.cc:491] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.466382 1984 raft_consensus.cc:513] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.466934 1984 leader_election.cc:290] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:02.467361 1166 ts_tablet_manager.cc:1939] T daac99da5d10475c9de263bef27baeb7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:02.468564 2370 raft_consensus.cc:513] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.468997 1160 tablet_replica.cc:331] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:02.469791 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fb1017fe6c4f5abd640ad1f6214232" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:02.469838 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet daac99da5d10475c9de263bef27baeb7 (table default.Table_d70feebb4775482eb8b3b251c50ad92c [id=27a0981b71084436994f1e9689afb145]) successfully deleted
I20250109 20:53:02.469967 775 raft_consensus.cc:3054] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:02.470957 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8d8220072e9f42ef90eda39e2382d2df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d8220072e9f42ef90eda39e2382d2df already in progress: creating tablet
I20250109 20:53:02.471213 1160 raft_consensus.cc:2238] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.471375 1160 raft_consensus.cc:2267] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:02.471916 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8f77f35085ff43a5b7fb6cce7e7a3409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f77f35085ff43a5b7fb6cce7e7a3409 already in progress: creating tablet
I20250109 20:53:02.472218 775 raft_consensus.cc:2463] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:02.472739 990 leader_election.cc:304] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:02.472885 1160 ts_tablet_manager.cc:1905] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.472975 2368 raft_consensus.cc:2798] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:02.473064 2368 raft_consensus.cc:695] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
W20250109 20:53:02.473577 713 leader_election.cc:343] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 825220629faa4d2481e0db9fa226c306
I20250109 20:53:02.474045 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 0f502d666aa043e2a0b67a4e21bcf03e (table default.Table_b086cd04a8e249a09d729be46eda67b1 [id=6aa6003be0684dc88b8ed001eed73727]) successfully deleted
I20250109 20:53:02.474025 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825220629faa4d2481e0db9fa226c306" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:02.474263 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f8232ec65c5f499896e68d118b3c4d79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8232ec65c5f499896e68d118b3c4d79 already in progress: creating tablet
W20250109 20:53:02.474447 712 leader_election.cc:343] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.474535 712 leader_election.cc:304] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:02.474830 1984 raft_consensus.cc:2743] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.474824 883 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0f27e5506444fd996d9ded42dfacb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.475168 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fb1017fe6c4f5abd640ad1f6214232" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:02.477494 989 leader_election.cc:343] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.481927 1158 tablet_bootstrap.cc:492] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.482048 1158 ts_tablet_manager.cc:1397] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.004s
I20250109 20:53:02.482847 1158 raft_consensus.cc:357] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.483011 1158 raft_consensus.cc:383] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.483060 1158 raft_consensus.cc:738] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.483238 1158 consensus_queue.cc:260] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.483639 1158 ts_tablet_manager.cc:1428] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:02.487373 1158 tablet_bootstrap.cc:492] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.490437 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577] alter complete (version 1)
I20250109 20:53:02.491186 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 2db4e85cd4094cf0b5b699402346b980 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:02.491405 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.491451 746 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.491614 1157 tablet_replica.cc:331] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:02.491698 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 90bcfd699e094c499df625281e7119b1 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:02.491781 1157 raft_consensus.cc:2238] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.491896 1157 raft_consensus.cc:2267] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.492507 1165 tablet_bootstrap.cc:492] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.492622 1165 ts_tablet_manager.cc:1397] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.149s user 0.013s sys 0.022s
I20250109 20:53:02.493304 1165 raft_consensus.cc:357] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.493495 1165 raft_consensus.cc:383] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.493544 1165 raft_consensus.cc:738] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.493731 1165 consensus_queue.cc:260] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.494199 1165 ts_tablet_manager.cc:1428] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:02.503897 1165 tablet_bootstrap.cc:492] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:02.472467 1166 tablet_replica.cc:331] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:02.505323 1165 tablet_bootstrap.cc:654] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.506103 1166 raft_consensus.cc:2238] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.506299 1166 raft_consensus.cc:2267] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.506794 1166 ts_tablet_manager.cc:1905] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.507259 1157 ts_tablet_manager.cc:1905] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.510401 1158 tablet_bootstrap.cc:654] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:02.516973 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b53c48fd0e0146cebf19d62519d1f773 consensus state has no leader
I20250109 20:53:02.518675 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_76d70836419947c78b06922a4fe21a2c" } new_table_name: "default.Table_2f66fee21e924800ac7afe7a12efb1eb" modify_external_catalogs: true
I20250109 20:53:02.521664 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7dacb59e95464e77b88e68386776f17d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.522156 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dacb59e95464e77b88e68386776f17d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.522560 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f2388e84f29e48a9b66925c5271adcdb" } new_table_name: "default.Table_2dc990ff61804866a5a8ea9d127694e9" modify_external_catalogs: true
I20250109 20:53:02.523542 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e1e8754625463ea42a92ef65a89214 with delete_type TABLET_DATA_DELETED (replaced by tablet 85a95a31468b431fb4304cf6680016da) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.524775 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7db9b887bea4440291c4923dbbe9e58e" } new_table_name: "default.Table_1d2a15480fd94e45985d90ef72fe9070" modify_external_catalogs: true
I20250109 20:53:02.536319 1158 tablet_bootstrap.cc:492] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.537626 1158 ts_tablet_manager.cc:1397] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.001s
I20250109 20:53:02.543468 2370 leader_election.cc:290] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:02.544773 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fb1017fe6c4f5abd640ad1f6214232" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:02.543937 1158 raft_consensus.cc:357] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.545807 1158 raft_consensus.cc:383] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.545867 1158 raft_consensus.cc:738] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.547406 2032 raft_consensus.cc:491] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.547523 2032 raft_consensus.cc:513] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.548022 2032 leader_election.cc:290] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:02.474647 2368 consensus_queue.cc:237] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:02.549461 854 leader_election.cc:343] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: aa7d7695ff5946abb6ee576431d2e87f
W20250109 20:53:02.549628 854 leader_election.cc:343] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: aa7d7695ff5946abb6ee576431d2e87f
I20250109 20:53:02.549696 854 leader_election.cc:304] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:02.550246 2032 raft_consensus.cc:2743] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:02.544656 989 consensus_peers.cc:487] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250109 20:53:02.550593 989 leader_election.cc:343] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.548995 1158 consensus_queue.cc:260] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.551769 1158 ts_tablet_manager.cc:1428] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250109 20:53:02.554400 1157 ts_tablet_manager.cc:1918] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.556488 1157 log.cc:1198] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/59ebeffe516c4c239ae2ee9040276e53
I20250109 20:53:02.556967 1157 ts_tablet_manager.cc:1939] T 59ebeffe516c4c239ae2ee9040276e53 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:02.560773 1166 ts_tablet_manager.cc:1918] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.560876 1166 log.cc:1198] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/0cede2a1882946a181b18d66f12c5114
I20250109 20:53:02.561131 1166 ts_tablet_manager.cc:1939] T 0cede2a1882946a181b18d66f12c5114 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:02.562196 1166 tablet_replica.cc:331] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:02.562348 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 0cede2a1882946a181b18d66f12c5114 (table default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]) successfully deleted
I20250109 20:53:02.562395 1166 raft_consensus.cc:2238] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.562510 1166 raft_consensus.cc:2267] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:02.562590 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4f1499e571354faf9952af5288b5c3fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1499e571354faf9952af5288b5c3fa already in progress: creating tablet
W20250109 20:53:02.562810 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a0bcc8bdce941d5a0df1f6d57dd2050 already in progress: creating tablet
I20250109 20:53:02.562970 1166 ts_tablet_manager.cc:1905] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.564939 1158 tablet_bootstrap.cc:492] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.566381 1158 tablet_bootstrap.cc:654] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.574478 1176 tablet_bootstrap.cc:492] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:02.578534 1176 ts_tablet_manager.cc:1397] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.125s user 0.001s sys 0.006s
I20250109 20:53:02.580361 1176 raft_consensus.cc:357] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.584653 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 59ebeffe516c4c239ae2ee9040276e53 (table default.Table_887acb86017941c4b8636c1008d835d9 [id=facd2e83422c492aa4337beff9ceb548]) successfully deleted
I20250109 20:53:02.585211 1157 tablet_replica.cc:331] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:02.585453 1157 raft_consensus.cc:2238] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.585659 1157 raft_consensus.cc:2267] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.586208 1157 ts_tablet_manager.cc:1905] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.589375 1176 raft_consensus.cc:383] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.589771 1160 ts_tablet_manager.cc:1918] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:02.589993 1176 raft_consensus.cc:738] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.593276 1176 consensus_queue.cc:260] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.593709 1160 log.cc:1198] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/0463dd5de987418d8ea1b102b09137bf
I20250109 20:53:02.598173 1160 ts_tablet_manager.cc:1939] T 0463dd5de987418d8ea1b102b09137bf P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:02.589774 1166 ts_tablet_manager.cc:1918] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.598527 1166 log.cc:1198] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/da7f67026b164e0faad128d34188afb5
I20250109 20:53:02.598899 1166 ts_tablet_manager.cc:1939] T da7f67026b164e0faad128d34188afb5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:02.599918 1166 tablet_replica.cc:331] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:02.600107 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet da7f67026b164e0faad128d34188afb5 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]) successfully deleted
I20250109 20:53:02.600150 1166 raft_consensus.cc:2238] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.600281 1166 raft_consensus.cc:2267] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.600775 1166 ts_tablet_manager.cc:1905] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.607115 1166 ts_tablet_manager.cc:1918] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.607227 1166 log.cc:1198] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/82209a6b39bb48e791ff63a8167c9765
I20250109 20:53:02.607527 1166 ts_tablet_manager.cc:1939] T 82209a6b39bb48e791ff63a8167c9765 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:02.608498 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 82209a6b39bb48e791ff63a8167c9765 (table default.Table_06b47da1327e425989c115839e9aecbe [id=cced112727b0498f955953f2b2d29551]) successfully deleted
I20250109 20:53:02.608952 1176 ts_tablet_manager.cc:1428] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250109 20:53:02.614171 1158 tablet_bootstrap.cc:492] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.614370 1158 ts_tablet_manager.cc:1397] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.012s
I20250109 20:53:02.615054 1158 raft_consensus.cc:357] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.615989 1165 tablet_bootstrap.cc:492] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.616151 1165 ts_tablet_manager.cc:1397] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.112s user 0.012s sys 0.014s
I20250109 20:53:02.617009 1165 raft_consensus.cc:357] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.617256 1165 raft_consensus.cc:383] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.617340 1165 raft_consensus.cc:738] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.617606 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_487b0b221267468ba9603345f5030c8b [id=50837eb593734f66a03e29d8edbc63a4]) successfully deleted
I20250109 20:53:02.617583 1165 consensus_queue.cc:260] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.618023 1165 ts_tablet_manager.cc:1428] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:02.618247 1165 tablet_bootstrap.cc:492] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:02.619791 1165 tablet_bootstrap.cc:654] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.620522 1158 raft_consensus.cc:383] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.620647 1158 raft_consensus.cc:738] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.620949 1158 consensus_queue.cc:260] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.621579 1158 ts_tablet_manager.cc:1428] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250109 20:53:02.621877 1158 tablet_bootstrap.cc:492] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.623142 1158 tablet_bootstrap.cc:654] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.633383 1165 tablet_bootstrap.cc:492] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.633515 1165 ts_tablet_manager.cc:1397] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20250109 20:53:02.634160 1165 raft_consensus.cc:357] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.634327 1165 raft_consensus.cc:383] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.634377 1165 raft_consensus.cc:738] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.634560 1165 consensus_queue.cc:260] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.635000 1165 ts_tablet_manager.cc:1428] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:02.635267 1165 tablet_bootstrap.cc:492] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:02.636704 1165 tablet_bootstrap.cc:654] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.638976 883 tablet_service.cc:1467] Processing CreateTablet for tablet 9f38f01144874befa5c38a7f6af2d1be (DEFAULT_TABLE table=default.Table_b67ae256dec74f84a6a39891b8e2ff60 [id=6270a0ad96824f8680808ca74e4ebc9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.639355 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f38f01144874befa5c38a7f6af2d1be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:02.648119 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36bc408ae5174be89dbc2a13de464dd4 consensus state has no leader
W20250109 20:53:02.658658 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70f06752b26c47f49d8486adc6c4d92c consensus state has no leader
W20250109 20:53:02.665508 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2dfbf64b178841d7aaf34632cbff3716 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dfbf64b178841d7aaf34632cbff3716 already in progress: creating tablet
I20250109 20:53:02.665920 1160 tablet_replica.cc:331] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:02.666203 1160 raft_consensus.cc:2238] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.666435 1160 raft_consensus.cc:2267] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.667007 1160 ts_tablet_manager.cc:1905] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.670253 1176 tablet_bootstrap.cc:492] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:02.671823 746 tablet_service.cc:1467] Processing CreateTablet for tablet 9f38f01144874befa5c38a7f6af2d1be (DEFAULT_TABLE table=default.Table_b67ae256dec74f84a6a39891b8e2ff60 [id=6270a0ad96824f8680808ca74e4ebc9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.672083 1176 tablet_bootstrap.cc:654] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.672222 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f38f01144874befa5c38a7f6af2d1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.673344 742 tablet_service.cc:1467] Processing CreateTablet for tablet 0a85def613874f98a0c9566c5b0a35a0 (DEFAULT_TABLE table=default.Table_0861a5b375f1434abfc4c91f0ae5dd2c [id=9dde810bad184c30b958e8229fe1f5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.673652 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a85def613874f98a0c9566c5b0a35a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.675242 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 65cd60f07e5f4cd098c1b54ea9f297fe (DEFAULT_TABLE table=default.Table_5bc556c741f74a51ac51116ee71ec9cd [id=fd30ce621d8d4205aaa42783d125a8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.677678 756 tablet_service.cc:1467] Processing CreateTablet for tablet 65cd60f07e5f4cd098c1b54ea9f297fe (DEFAULT_TABLE table=default.Table_5bc556c741f74a51ac51116ee71ec9cd [id=fd30ce621d8d4205aaa42783d125a8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.678795 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 9f38f01144874befa5c38a7f6af2d1be (DEFAULT_TABLE table=default.Table_b67ae256dec74f84a6a39891b8e2ff60 [id=6270a0ad96824f8680808ca74e4ebc9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.679181 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 0a85def613874f98a0c9566c5b0a35a0 (DEFAULT_TABLE table=default.Table_0861a5b375f1434abfc4c91f0ae5dd2c [id=9dde810bad184c30b958e8229fe1f5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.679486 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65cd60f07e5f4cd098c1b54ea9f297fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.679807 881 tablet_service.cc:1467] Processing CreateTablet for tablet 65cd60f07e5f4cd098c1b54ea9f297fe (DEFAULT_TABLE table=default.Table_5bc556c741f74a51ac51116ee71ec9cd [id=fd30ce621d8d4205aaa42783d125a8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.680119 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65cd60f07e5f4cd098c1b54ea9f297fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.697396 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0a85def613874f98a0c9566c5b0a35a0 (DEFAULT_TABLE table=default.Table_0861a5b375f1434abfc4c91f0ae5dd2c [id=9dde810bad184c30b958e8229fe1f5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.698145 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a85def613874f98a0c9566c5b0a35a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.703837 895 tablet_service.cc:1467] Processing CreateTablet for tablet 86926f666672467498b989e91fa04607 (DEFAULT_TABLE table=default.Table_31e530948ce441d0bfa57b94cd6df55d [id=9f1f24004bab440197504a7bf17f5b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.704185 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86926f666672467498b989e91fa04607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.707139 2368 raft_consensus.cc:491] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.707291 2368 raft_consensus.cc:513] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.707953 2368 leader_election.cc:290] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:02.708442 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e716270ffcf44039b075023fd0c2d2a3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:02.708986 990 leader_election.cc:343] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.709414 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e716270ffcf44039b075023fd0c2d2a3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:02.709694 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 1181f2e015334972914ac266d51ac537 (DEFAULT_TABLE table=default.Table_b263bd08e0584027b3c4534c8d423710 [id=13456b3ad03443ccb90a307c665c1111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.712419 888 tablet_service.cc:1467] Processing CreateTablet for tablet df88f0ecefae4e7e92db39af76bab76f (DEFAULT_TABLE table=default.Table_4453facc7e5d4f2f8f7cacf989aff0fc [id=5ac2952eda7e4022838583aa3e5d1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.712692 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df88f0ecefae4e7e92db39af76bab76f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:02.717058 989 leader_election.cc:343] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.717166 989 leader_election.cc:304] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:02.717764 2368 raft_consensus.cc:2743] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.720127 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 86926f666672467498b989e91fa04607 (DEFAULT_TABLE table=default.Table_31e530948ce441d0bfa57b94cd6df55d [id=9f1f24004bab440197504a7bf17f5b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.720454 746 tablet_service.cc:1467] Processing CreateTablet for tablet 86926f666672467498b989e91fa04607 (DEFAULT_TABLE table=default.Table_31e530948ce441d0bfa57b94cd6df55d [id=9f1f24004bab440197504a7bf17f5b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.721083 756 tablet_service.cc:1467] Processing CreateTablet for tablet fc749eddce6444ee82a1846a19e5c039 (DEFAULT_TABLE table=default.Table_c18d342c6c2f416e88d8e3049d4909f1 [id=212228b865f64eabad675a09ecb730a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.721448 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc749eddce6444ee82a1846a19e5c039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.722010 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1181f2e015334972914ac266d51ac537 (DEFAULT_TABLE table=default.Table_b263bd08e0584027b3c4534c8d423710 [id=13456b3ad03443ccb90a307c665c1111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.722400 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1181f2e015334972914ac266d51ac537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.723115 1017 tablet_service.cc:1467] Processing CreateTablet for tablet df88f0ecefae4e7e92db39af76bab76f (DEFAULT_TABLE table=default.Table_4453facc7e5d4f2f8f7cacf989aff0fc [id=5ac2952eda7e4022838583aa3e5d1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.724310 747 tablet_service.cc:1467] Processing CreateTablet for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 (DEFAULT_TABLE table=default.Table_8aa7a8f99b5a4c5293df93fcbd7ba167 [id=fddb27804add4041b2c7ee7f131a895f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.724558 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.724851 1019 tablet_service.cc:1467] Processing CreateTablet for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 (DEFAULT_TABLE table=default.Table_8aa7a8f99b5a4c5293df93fcbd7ba167 [id=fddb27804add4041b2c7ee7f131a895f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.724879 750 tablet_service.cc:1467] Processing CreateTablet for tablet df88f0ecefae4e7e92db39af76bab76f (DEFAULT_TABLE table=default.Table_4453facc7e5d4f2f8f7cacf989aff0fc [id=5ac2952eda7e4022838583aa3e5d1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.725142 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df88f0ecefae4e7e92db39af76bab76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.725939 888 tablet_service.cc:1467] Processing CreateTablet for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 (DEFAULT_TABLE table=default.Table_8aa7a8f99b5a4c5293df93fcbd7ba167 [id=fddb27804add4041b2c7ee7f131a895f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.726270 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.726459 745 tablet_service.cc:1467] Processing CreateTablet for tablet c6b18c3ad3814ae38b4aa70a1157dfee (DEFAULT_TABLE table=default.Table_6da7024fb0764c07a7d8ef603e5196f6 [id=6cb9107dea944907a8c175700e1d16f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.726697 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6b18c3ad3814ae38b4aa70a1157dfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.727867 890 tablet_service.cc:1467] Processing CreateTablet for tablet c6b18c3ad3814ae38b4aa70a1157dfee (DEFAULT_TABLE table=default.Table_6da7024fb0764c07a7d8ef603e5196f6 [id=6cb9107dea944907a8c175700e1d16f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.728108 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6b18c3ad3814ae38b4aa70a1157dfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.728907 1016 tablet_service.cc:1467] Processing CreateTablet for tablet fc749eddce6444ee82a1846a19e5c039 (DEFAULT_TABLE table=default.Table_c18d342c6c2f416e88d8e3049d4909f1 [id=212228b865f64eabad675a09ecb730a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.729348 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86926f666672467498b989e91fa04607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.729982 881 tablet_service.cc:1467] Processing CreateTablet for tablet 1181f2e015334972914ac266d51ac537 (DEFAULT_TABLE table=default.Table_b263bd08e0584027b3c4534c8d423710 [id=13456b3ad03443ccb90a307c665c1111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.730242 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1181f2e015334972914ac266d51ac537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.731359 2368 raft_consensus.cc:491] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.731455 2368 raft_consensus.cc:513] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.732038 2368 leader_election.cc:290] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:02.732632 1030 tablet_service.cc:1467] Processing CreateTablet for tablet c6b18c3ad3814ae38b4aa70a1157dfee (DEFAULT_TABLE table=default.Table_6da7024fb0764c07a7d8ef603e5196f6 [id=6cb9107dea944907a8c175700e1d16f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.732900 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006475691733470f8a0468365972dfd2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:02.732900 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006475691733470f8a0468365972dfd2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:02.733321 990 leader_election.cc:343] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:02.733434 989 leader_election.cc:343] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.733490 989 leader_election.cc:304] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:02.733888 2368 raft_consensus.cc:2743] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.735361 882 tablet_service.cc:1467] Processing CreateTablet for tablet db0ccfe9ad0f44078fe1fe32cec1560c (DEFAULT_TABLE table=default.Table_2ebffe648ce34afaa688cf8cb881e20f [id=b503811c9930437da7f9352a35e9b19a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.735653 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0ccfe9ad0f44078fe1fe32cec1560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.737577 1023 tablet_service.cc:1467] Processing CreateTablet for tablet db0ccfe9ad0f44078fe1fe32cec1560c (DEFAULT_TABLE table=default.Table_2ebffe648ce34afaa688cf8cb881e20f [id=b503811c9930437da7f9352a35e9b19a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.737993 750 tablet_service.cc:1467] Processing CreateTablet for tablet db0ccfe9ad0f44078fe1fe32cec1560c (DEFAULT_TABLE table=default.Table_2ebffe648ce34afaa688cf8cb881e20f [id=b503811c9930437da7f9352a35e9b19a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.738276 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0ccfe9ad0f44078fe1fe32cec1560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.741865 895 tablet_service.cc:1467] Processing CreateTablet for tablet fc749eddce6444ee82a1846a19e5c039 (DEFAULT_TABLE table=default.Table_c18d342c6c2f416e88d8e3049d4909f1 [id=212228b865f64eabad675a09ecb730a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.742197 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc749eddce6444ee82a1846a19e5c039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.748816 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f38f01144874befa5c38a7f6af2d1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.753078 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a85def613874f98a0c9566c5b0a35a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.756999 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1181f2e015334972914ac266d51ac537. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.757162 882 tablet_service.cc:1467] Processing CreateTablet for tablet 34ca0a49aef34c40982ff34464727a5e (DEFAULT_TABLE table=default.Table_ee5424d3946544f49aa8a1f94a0fbbc2 [id=349aa98736f344f2a7b4c7339a836756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.758114 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ca0a49aef34c40982ff34464727a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.758074 887 tablet_service.cc:1467] Processing CreateTablet for tablet eada37af2bc446d48531864e8f3404f9 (DEFAULT_TABLE table=default.Table_9d6b86869c9849318edb0e8e4360f246 [id=82b4e8fc936e453288142f38318095a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.758332 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eada37af2bc446d48531864e8f3404f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.760963 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86926f666672467498b989e91fa04607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.761155 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.761327 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc749eddce6444ee82a1846a19e5c039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.761515 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6b18c3ad3814ae38b4aa70a1157dfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.761953 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0ccfe9ad0f44078fe1fe32cec1560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.764837 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df88f0ecefae4e7e92db39af76bab76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.774089 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 5d841529b90145caad4c4fd19f9c9669 (DEFAULT_TABLE table=default.Table_21ed68ee9ed84331ae175e7fe5d6bd6f [id=5558772702244188937885f3a5328074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.774430 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d841529b90145caad4c4fd19f9c9669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.778923 746 tablet_service.cc:1467] Processing CreateTablet for tablet 5d841529b90145caad4c4fd19f9c9669 (DEFAULT_TABLE table=default.Table_21ed68ee9ed84331ae175e7fe5d6bd6f [id=5558772702244188937885f3a5328074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.779119 1166 tablet_replica.cc:331] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:02.779305 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d841529b90145caad4c4fd19f9c9669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.779292 1166 raft_consensus.cc:2238] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:02.779580 1166 raft_consensus.cc:2267] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.780084 1166 ts_tablet_manager.cc:1905] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.781193 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 34ca0a49aef34c40982ff34464727a5e (DEFAULT_TABLE table=default.Table_ee5424d3946544f49aa8a1f94a0fbbc2 [id=349aa98736f344f2a7b4c7339a836756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.781541 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ca0a49aef34c40982ff34464727a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.784937 756 tablet_service.cc:1467] Processing CreateTablet for tablet eada37af2bc446d48531864e8f3404f9 (DEFAULT_TABLE table=default.Table_9d6b86869c9849318edb0e8e4360f246 [id=82b4e8fc936e453288142f38318095a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.785279 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eada37af2bc446d48531864e8f3404f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.790603 750 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2c4a86b3ba43fdade03d9697b0e9f2 (DEFAULT_TABLE table=default.Table_08bc5abe888e442ba8afb7fe0b9d746a [id=1bd280cab8484a259f4143dc928a72ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.790908 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2c4a86b3ba43fdade03d9697b0e9f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:02.793015 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 733c7059c750463aa586d9b733f1d1e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 733c7059c750463aa586d9b733f1d1e9 already in progress: creating tablet
W20250109 20:53:02.793298 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ceb2a882918046af95a884b87b0d084f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceb2a882918046af95a884b87b0d084f already in progress: creating tablet
W20250109 20:53:02.793737 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f8232ec65c5f499896e68d118b3c4d79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8232ec65c5f499896e68d118b3c4d79 already in progress: creating tablet
I20250109 20:53:02.794279 747 tablet_service.cc:1467] Processing CreateTablet for tablet 34ca0a49aef34c40982ff34464727a5e (DEFAULT_TABLE table=default.Table_ee5424d3946544f49aa8a1f94a0fbbc2 [id=349aa98736f344f2a7b4c7339a836756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.794636 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5d841529b90145caad4c4fd19f9c9669 (DEFAULT_TABLE table=default.Table_21ed68ee9ed84331ae175e7fe5d6bd6f [id=5558772702244188937885f3a5328074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.794986 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ca0a49aef34c40982ff34464727a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.795297 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2c4a86b3ba43fdade03d9697b0e9f2 (DEFAULT_TABLE table=default.Table_08bc5abe888e442ba8afb7fe0b9d746a [id=1bd280cab8484a259f4143dc928a72ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.795698 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d841529b90145caad4c4fd19f9c9669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.797643 1165 tablet_bootstrap.cc:492] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.797792 1165 ts_tablet_manager.cc:1397] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.163s user 0.011s sys 0.023s
I20250109 20:53:02.798539 1165 raft_consensus.cc:357] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.798707 1165 raft_consensus.cc:383] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.798758 1165 raft_consensus.cc:738] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.798933 1165 consensus_queue.cc:260] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.799432 1165 ts_tablet_manager.cc:1428] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:02.799669 1165 tablet_bootstrap.cc:492] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:02.803602 1165 tablet_bootstrap.cc:654] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.803892 750 tablet_service.cc:1467] Processing CreateTablet for tablet 0c49e78725c548c7ad4b61d9632e8931 (DEFAULT_TABLE table=default.Table_311b8a2f631245c9979c5be620fd1791 [id=0533908885734c8e8698ba5abcf0db8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.804250 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c49e78725c548c7ad4b61d9632e8931. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.806007 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65cd60f07e5f4cd098c1b54ea9f297fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.807224 756 tablet_service.cc:1467] Processing CreateTablet for tablet 417aab64aaef487a83c1e0ff6eec9ffc (DEFAULT_TABLE table=default.Table_8e249bfdf7084d0d838c0c7b47e75609 [id=b331038542344e6d959280c9d2bbf3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.807492 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417aab64aaef487a83c1e0ff6eec9ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.812794 746 tablet_service.cc:1467] Processing CreateTablet for tablet 908781d94a054386b7d7c06c2c9fa3ca (DEFAULT_TABLE table=default.Table_1a2eba2c16714ea1951f0037668ef8f4 [id=86dbc4908e104814a0a5c7198347e2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.813225 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908781d94a054386b7d7c06c2c9fa3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.815142 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cb96e72193f94850b49ec5ac33b2cbe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:02.815474 1157 ts_tablet_manager.cc:1918] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.815557 1157 log.cc:1198] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/bb0d889cb1114edc8b799b41b23a54a6
W20250109 20:53:02.815728 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb96e72193f94850b49ec5ac33b2cbe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.815834 1157 ts_tablet_manager.cc:1939] T bb0d889cb1114edc8b799b41b23a54a6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:02.817018 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_45be879858bb4f5c9c9513bec5a7d2ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.817458 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45be879858bb4f5c9c9513bec5a7d2ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.807071 887 tablet_service.cc:1467] Processing CreateTablet for tablet 417aab64aaef487a83c1e0ff6eec9ffc (DEFAULT_TABLE table=default.Table_8e249bfdf7084d0d838c0c7b47e75609 [id=b331038542344e6d959280c9d2bbf3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.823704 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417aab64aaef487a83c1e0ff6eec9ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.823719 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet bb0d889cb1114edc8b799b41b23a54a6 (table default.Table_0bef1709ef9c47738d52c3d41d4f6819 [id=65098b4e94ce440494b2d5682132ba1a]) successfully deleted
I20250109 20:53:02.829316 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0ac6f8baab5040abbe36e185a8052c6e
I20250109 20:53:02.810581 883 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2c4a86b3ba43fdade03d9697b0e9f2 (DEFAULT_TABLE table=default.Table_08bc5abe888e442ba8afb7fe0b9d746a [id=1bd280cab8484a259f4143dc928a72ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.830093 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2c4a86b3ba43fdade03d9697b0e9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.831589 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac6f8baab5040abbe36e185a8052c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.832468 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac6f8baab5040abbe36e185a8052c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.833199 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ab7d4f31211742e3b23e9f22e36cf7c7
I20250109 20:53:02.833701 1158 tablet_bootstrap.cc:492] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.833874 1158 ts_tablet_manager.cc:1397] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.212s user 0.014s sys 0.025s
I20250109 20:53:02.834637 1158 raft_consensus.cc:357] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.834867 1158 raft_consensus.cc:383] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.834954 1158 raft_consensus.cc:738] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.835230 1158 consensus_queue.cc:260] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.835546 895 tablet_service.cc:1514] Processing DeleteTablet for tablet aa7d7695ff5946abb6ee576431d2e87f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.835845 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7d4f31211742e3b23e9f22e36cf7c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:02.836313 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d3a253685e34e80a26f338be66d31bb consensus state has no leader
I20250109 20:53:02.836808 887 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7d4f31211742e3b23e9f22e36cf7c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.837877 1158 ts_tablet_manager.cc:1428] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:02.841140 742 tablet_service.cc:1467] Processing CreateTablet for tablet c29b4c6866834879b64b80b9f4655799 (DEFAULT_TABLE table=default.Table_f60d07bbecfd4e478a06f79f46ba125a [id=df7f0fa63a8647639f06e3e713d9cb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.841627 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29b4c6866834879b64b80b9f4655799. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.842217 895 tablet_service.cc:1467] Processing CreateTablet for tablet a7aa8b2c8e744aa3a1ac70fef238e65b (DEFAULT_TABLE table=default.Table_7bb113fc7cc443749bc8851afc2fc51e [id=55d851980ac349c3b3f00f3f58ddd346]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.842522 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7aa8b2c8e744aa3a1ac70fef238e65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.843477 1158 tablet_bootstrap.cc:492] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.844975 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:53:02.846267 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_615f5ae1a3c84fbd85db081530862f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.846765 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_615f5ae1a3c84fbd85db081530862f20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.847941 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c8cea69ded7f4343949b6bd1b8ab7b1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.848357 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8cea69ded7f4343949b6bd1b8ab7b1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.849449 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed00f7954143c9ba9bf65d8a8a68d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.847921 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4c067bae363a4fdfa0fd5de56862c35c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.850292 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c067bae363a4fdfa0fd5de56862c35c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.854143 887 tablet_service.cc:1467] Processing CreateTablet for tablet c29b4c6866834879b64b80b9f4655799 (DEFAULT_TABLE table=default.Table_f60d07bbecfd4e478a06f79f46ba125a [id=df7f0fa63a8647639f06e3e713d9cb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.804507 1019 tablet_service.cc:1467] Processing CreateTablet for tablet eada37af2bc446d48531864e8f3404f9 (DEFAULT_TABLE table=default.Table_9d6b86869c9849318edb0e8e4360f246 [id=82b4e8fc936e453288142f38318095a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.803892 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 0c49e78725c548c7ad4b61d9632e8931 (DEFAULT_TABLE table=default.Table_311b8a2f631245c9979c5be620fd1791 [id=0533908885734c8e8698ba5abcf0db8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.808326 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 417aab64aaef487a83c1e0ff6eec9ffc (DEFAULT_TABLE table=default.Table_8e249bfdf7084d0d838c0c7b47e75609 [id=b331038542344e6d959280c9d2bbf3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.810129 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 908781d94a054386b7d7c06c2c9fa3ca (DEFAULT_TABLE table=default.Table_1a2eba2c16714ea1951f0037668ef8f4 [id=86dbc4908e104814a0a5c7198347e2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.832021 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac6f8baab5040abbe36e185a8052c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.835031 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7d4f31211742e3b23e9f22e36cf7c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.841609 1032 tablet_service.cc:1467] Processing CreateTablet for tablet c29b4c6866834879b64b80b9f4655799 (DEFAULT_TABLE table=default.Table_f60d07bbecfd4e478a06f79f46ba125a [id=df7f0fa63a8647639f06e3e713d9cb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.842716 1176 tablet_bootstrap.cc:492] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:02.858180 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29b4c6866834879b64b80b9f4655799. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.858335 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908781d94a054386b7d7c06c2c9fa3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.858453 750 tablet_service.cc:1467] Processing CreateTablet for tablet a7aa8b2c8e744aa3a1ac70fef238e65b (DEFAULT_TABLE table=default.Table_7bb113fc7cc443749bc8851afc2fc51e [id=55d851980ac349c3b3f00f3f58ddd346]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.859258 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2db4e85cd4094cf0b5b699402346b980
I20250109 20:53:02.859684 449 catalog_manager.cc:6596] ReplaceTablet: tablet 2db4e85cd4094cf0b5b699402346b980 deleted and replaced by tablet b54b66ca10114330b3485f67eb076064
I20250109 20:53:02.859958 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b263bd08e0584027b3c4534c8d423710" } new_table_name: "default.Table_e61c9814e46a4d32994bdc0e5eaa23ee" modify_external_catalogs: true
I20250109 20:53:02.860833 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db4e85cd4094cf0b5b699402346b980 with delete_type TABLET_DATA_DELETED (replaced by tablet b54b66ca10114330b3485f67eb076064) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.860929 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417aab64aaef487a83c1e0ff6eec9ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.861227 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db4e85cd4094cf0b5b699402346b980 with delete_type TABLET_DATA_DELETED (replaced by tablet b54b66ca10114330b3485f67eb076064) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.861550 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2c4a86b3ba43fdade03d9697b0e9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.862695 446 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]). Expected version 1 got 0
I20250109 20:53:02.863061 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 14fb1017fe6c4f5abd640ad1f6214232
I20250109 20:53:02.863587 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b67ae256dec74f84a6a39891b8e2ff60" } modify_external_catalogs: true
I20250109 20:53:02.863734 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b67ae256dec74f84a6a39891b8e2ff60" } modify_external_catalogs: true
I20250109 20:53:02.865707 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c49e78725c548c7ad4b61d9632e8931. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.865926 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fb1017fe6c4f5abd640ad1f6214232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.868396 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fb1017fe6c4f5abd640ad1f6214232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.869174 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fb1017fe6c4f5abd640ad1f6214232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.869248 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eada37af2bc446d48531864e8f3404f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.821787 881 tablet_service.cc:1467] Processing CreateTablet for tablet 0c49e78725c548c7ad4b61d9632e8931 (DEFAULT_TABLE table=default.Table_311b8a2f631245c9979c5be620fd1791 [id=0533908885734c8e8698ba5abcf0db8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.875134 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1c0dbdff917446539c1bbfa0600d5bbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.875617 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c0dbdff917446539c1bbfa0600d5bbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.878247 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bc4cb5b8559b475fb49723c73456ce40" } modify_external_catalogs: true
I20250109 20:53:02.878377 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bc4cb5b8559b475fb49723c73456ce40" } modify_external_catalogs: true
I20250109 20:53:02.878801 1176 ts_tablet_manager.cc:1397] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.188s user 0.004s sys 0.002s
I20250109 20:53:02.879804 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 825220629faa4d2481e0db9fa226c306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.879801 746 tablet_service.cc:1514] Processing DeleteTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.881219 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9014025f02f448f7bad5dd2f9cd1d8d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:02.881664 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9014025f02f448f7bad5dd2f9cd1d8d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.882378 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet bd62a3ae64a143998bb1172b3a97c69f with delete_type TABLET_DATA_DELETED (replaced by tablet 575f3cb9f61143ab949fa85f47edcafb) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.884068 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7aa8b2c8e744aa3a1ac70fef238e65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.885233 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 006475691733470f8a0468365972dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.884068 1176 raft_consensus.cc:357] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.885717 1176 raft_consensus.cc:383] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250109 20:53:02.887380 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0ac6f8baab5040abbe36e185a8052c6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ac6f8baab5040abbe36e185a8052c6e already in progress: creating tablet
I20250109 20:53:02.886785 1176 raft_consensus.cc:738] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:53:02.887583 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ab7d4f31211742e3b23e9f22e36cf7c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab7d4f31211742e3b23e9f22e36cf7c7 already in progress: creating tablet
I20250109 20:53:02.887847 1158 tablet_bootstrap.cc:654] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.888311 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 90bcfd699e094c499df625281e7119b1
I20250109 20:53:02.888686 450 catalog_manager.cc:6596] ReplaceTablet: tablet 90bcfd699e094c499df625281e7119b1 deleted and replaced by tablet 70581bd3aa674b73ab1b4908fe1c9030
I20250109 20:53:02.888960 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 90bcfd699e094c499df625281e7119b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 70581bd3aa674b73ab1b4908fe1c9030) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.889407 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c29b4c6866834879b64b80b9f4655799. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.889737 1157 tablet_replica.cc:331] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:02.889237 1176 consensus_queue.cc:260] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.822252 888 tablet_service.cc:1467] Processing CreateTablet for tablet 908781d94a054386b7d7c06c2c9fa3ca (DEFAULT_TABLE table=default.Table_1a2eba2c16714ea1951f0037668ef8f4 [id=86dbc4908e104814a0a5c7198347e2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.891022 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908781d94a054386b7d7c06c2c9fa3ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:02.891597 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2db4e85cd4094cf0b5b699402346b980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2db4e85cd4094cf0b5b699402346b980 already in progress: creating tablet
I20250109 20:53:02.892576 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 90bcfd699e094c499df625281e7119b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 70581bd3aa674b73ab1b4908fe1c9030) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.892577 890 tablet_service.cc:1467] Processing CreateTablet for tablet b0dd5ca642034441a0eb5c5a653b7263 (DEFAULT_TABLE table=default.Table_38713c5cbbbf44ba95a5b4710cdfcd44 [id=cc58e9ed135c4f18a62a76ee845e72af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.892805 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.892904 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c49e78725c548c7ad4b61d9632e8931. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.893146 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0dd5ca642034441a0eb5c5a653b7263. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.895946 1030 tablet_service.cc:1467] Processing CreateTablet for tablet a7aa8b2c8e744aa3a1ac70fef238e65b (DEFAULT_TABLE table=default.Table_7bb113fc7cc443749bc8851afc2fc51e [id=55d851980ac349c3b3f00f3f58ddd346]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.896334 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7aa8b2c8e744aa3a1ac70fef238e65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.892803 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 90bcfd699e094c499df625281e7119b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 70581bd3aa674b73ab1b4908fe1c9030) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.855407 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8d67b624984a0bb327d045c55476da with delete_type TABLET_DATA_DELETED (replaced by tablet 687db3e734af47339c2546970a71a539) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.904784 2368 raft_consensus.cc:491] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.904910 2368 raft_consensus.cc:513] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.905126 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:53:02.905535 2368 leader_election.cc:290] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:02.906499 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:02.906594 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd758fa2fa4cebbdf7faf3856f1941) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:02.907219 989 leader_election.cc:343] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.908900 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:02.909551 990 leader_election.cc:343] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:02.909647 990 leader_election.cc:304] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:02.911048 2368 raft_consensus.cc:2743] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.912365 1157 raft_consensus.cc:2238] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.913744 750 tablet_service.cc:1467] Processing CreateTablet for tablet b0dd5ca642034441a0eb5c5a653b7263 (DEFAULT_TABLE table=default.Table_38713c5cbbbf44ba95a5b4710cdfcd44 [id=cc58e9ed135c4f18a62a76ee845e72af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.914266 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0dd5ca642034441a0eb5c5a653b7263. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.915043 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f2e044c26f7c4b9d985304d701432239" } new_table_name: "default.Table_a31ca0f1488740a29d53d908ab59b00a" modify_external_catalogs: true
I20250109 20:53:02.915057 1157 raft_consensus.cc:2267] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:02.916325 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9b3a6884c07847b090850ad3f39476db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:02.916345 1176 ts_tablet_manager.cc:1428] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
W20250109 20:53:02.916853 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b3a6884c07847b090850ad3f39476db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.918516 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:02.918488 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbbb16301714d0e941d268f93ae075b (DEFAULT_TABLE table=default.Table_9e2fb7942ca744539c839f031e3528ed [id=083bda70b2d64c65b122caa0815a9f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.918784 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbbb16301714d0e941d268f93ae075b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.919675 1165 tablet_bootstrap.cc:492] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.919783 1165 ts_tablet_manager.cc:1397] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.120s user 0.015s sys 0.016s
I20250109 20:53:02.920392 1165 raft_consensus.cc:357] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.920522 1165 raft_consensus.cc:383] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.920557 1165 raft_consensus.cc:738] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.920759 1165 consensus_queue.cc:260] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.921196 1165 ts_tablet_manager.cc:1428] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:02.921725 1165 tablet_bootstrap.cc:492] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:02.923373 1165 tablet_bootstrap.cc:654] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.929913 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbbb16301714d0e941d268f93ae075b (DEFAULT_TABLE table=default.Table_9e2fb7942ca744539c839f031e3528ed [id=083bda70b2d64c65b122caa0815a9f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.930275 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbbb16301714d0e941d268f93ae075b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.931759 1030 tablet_service.cc:1467] Processing CreateTablet for tablet b0dd5ca642034441a0eb5c5a653b7263 (DEFAULT_TABLE table=default.Table_38713c5cbbbf44ba95a5b4710cdfcd44 [id=cc58e9ed135c4f18a62a76ee845e72af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.932834 890 tablet_service.cc:1467] Processing CreateTablet for tablet e53966daf1ed4a67801d9ab239e17d8e (DEFAULT_TABLE table=default.Table_62fca2bdab804af8ab92b2eb60115336 [id=689bfafd8fd74c8a8ac06f9a8c68984d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.933223 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53966daf1ed4a67801d9ab239e17d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.937016 742 tablet_service.cc:1467] Processing CreateTablet for tablet e53966daf1ed4a67801d9ab239e17d8e (DEFAULT_TABLE table=default.Table_62fca2bdab804af8ab92b2eb60115336 [id=689bfafd8fd74c8a8ac06f9a8c68984d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.937428 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53966daf1ed4a67801d9ab239e17d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.938303 1165 tablet_bootstrap.cc:492] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.938457 1165 ts_tablet_manager.cc:1397] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20250109 20:53:02.939049 1157 ts_tablet_manager.cc:1905] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:02.939170 1165 raft_consensus.cc:357] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.939401 1165 raft_consensus.cc:383] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.939494 1165 raft_consensus.cc:738] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.939762 1165 consensus_queue.cc:260] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.940272 1165 ts_tablet_manager.cc:1428] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:02.940754 1165 tablet_bootstrap.cc:492] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:02.942230 1165 tablet_bootstrap.cc:654] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.814886 1160 ts_tablet_manager.cc:1918] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:02.944903 1160 log.cc:1198] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/06c3aa20539740cba4e5d6e852ad2909
I20250109 20:53:02.945345 1160 ts_tablet_manager.cc:1939] T 06c3aa20539740cba4e5d6e852ad2909 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:02.946563 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 06c3aa20539740cba4e5d6e852ad2909 (table default.Table_5e3e729549234c39a601e29da97094d4 [id=5811deb4d7214be490d07cf674920448]) successfully deleted
I20250109 20:53:02.942551 1176 tablet_bootstrap.cc:492] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:02.946805 1160 tablet_replica.cc:331] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:02.946902 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0dd5ca642034441a0eb5c5a653b7263. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.946957 1160 raft_consensus.cc:2238] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:02.947093 1160 raft_consensus.cc:2267] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:02.947233 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0a2efcc2a6ec4b109845d8087491ca9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2efcc2a6ec4b109845d8087491ca9f already in progress: creating tablet
W20250109 20:53:02.947408 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d3428128a5494341b1986ce6956bb88a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3428128a5494341b1986ce6956bb88a already in progress: creating tablet
I20250109 20:53:02.947505 1160 ts_tablet_manager.cc:1905] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:02.947566 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ef0f27e5506444fd996d9ded42dfacb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef0f27e5506444fd996d9ded42dfacb7 already in progress: creating tablet
I20250109 20:53:02.948097 2032 raft_consensus.cc:491] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:02.948208 2032 raft_consensus.cc:513] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.948858 2032 leader_election.cc:290] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:02.948855 1176 tablet_bootstrap.cc:654] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.949445 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda300a7e58a44339c3bdb2e62a9f384" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:02.950284 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda300a7e58a44339c3bdb2e62a9f384" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:02.949875 888 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbbb16301714d0e941d268f93ae075b (DEFAULT_TABLE table=default.Table_9e2fb7942ca744539c839f031e3528ed [id=083bda70b2d64c65b122caa0815a9f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.950747 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db4e85cd4094cf0b5b699402346b980 with delete_type TABLET_DATA_DELETED (replaced by tablet b54b66ca10114330b3485f67eb076064) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.950836 854 leader_election.cc:304] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:02.951407 2032 raft_consensus.cc:2743] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:02.951884 1158 tablet_bootstrap.cc:492] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:02.952056 1158 ts_tablet_manager.cc:1397] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.002s
I20250109 20:53:02.952898 1158 raft_consensus.cc:357] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.953207 1158 raft_consensus.cc:383] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.953367 1158 raft_consensus.cc:738] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.953857 1158 consensus_queue.cc:260] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.954375 1166 ts_tablet_manager.cc:1918] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:02.954520 1166 log.cc:1198] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/0463dd5de987418d8ea1b102b09137bf
I20250109 20:53:02.955156 1166 ts_tablet_manager.cc:1939] T 0463dd5de987418d8ea1b102b09137bf P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:02.956959 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 009b90274d6b4830bf7402467f1672df (DEFAULT_TABLE table=default.Table_baf2e0ecb9594bc3af2f86ba420846d4 [id=8ebdd3108bfd4995905f2e90c646d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.957281 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009b90274d6b4830bf7402467f1672df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.959136 1158 ts_tablet_manager.cc:1428] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:53:02.960821 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 0463dd5de987418d8ea1b102b09137bf (table default.Table_487b0b221267468ba9603345f5030c8b [id=50837eb593734f66a03e29d8edbc63a4]) successfully deleted
I20250109 20:53:02.959867 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbbb16301714d0e941d268f93ae075b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.964962 1158 tablet_bootstrap.cc:492] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:02.965440 447 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 165f6304acc849569a8e90d135a8e2d0 (table default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320]). Expected version 1 got 0
I20250109 20:53:02.966538 1165 tablet_bootstrap.cc:492] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:02.966670 1165 ts_tablet_manager.cc:1397] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20250109 20:53:02.966929 895 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596614521a4a92a71b781b162f70b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.967257 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 2b1891cbef8f4b0cae0c15d478db4860 (table default.Table_ca51464e265c4e28b9e54fe7be0587fc [id=0a0599668d77405ba0e72eda2aa5219e]). Expected version 4 got 0
I20250109 20:53:02.967329 1165 raft_consensus.cc:357] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.967576 1165 raft_consensus.cc:383] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:02.967669 1165 raft_consensus.cc:738] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:02.967916 1165 consensus_queue.cc:260] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:02.968384 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_69a8a32a033547b28e591ec3684a668e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:02.968586 742 tablet_service.cc:1467] Processing CreateTablet for tablet 009b90274d6b4830bf7402467f1672df (DEFAULT_TABLE table=default.Table_baf2e0ecb9594bc3af2f86ba420846d4 [id=8ebdd3108bfd4995905f2e90c646d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.969520 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009b90274d6b4830bf7402467f1672df. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:02.961217 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2dfbf64b178841d7aaf34632cbff3716 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dfbf64b178841d7aaf34632cbff3716 already in progress: creating tablet
I20250109 20:53:02.971560 750 tablet_service.cc:1467] Processing CreateTablet for tablet bfa3e55fc83243a3b4978e981fdadf8c (DEFAULT_TABLE table=default.Table_908fd543e71a4471a5c0f402e7ac5a7a [id=0b0dd02c61654bf9831b408ca831f2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.971549 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7e4416ccc3ff49dab092f17ee96df2e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:02.971848 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa3e55fc83243a3b4978e981fdadf8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.971836 1032 tablet_service.cc:1467] Processing CreateTablet for tablet bfa3e55fc83243a3b4978e981fdadf8c (DEFAULT_TABLE table=default.Table_908fd543e71a4471a5c0f402e7ac5a7a [id=0b0dd02c61654bf9831b408ca831f2cd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:02.972119 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e4416ccc3ff49dab092f17ee96df2e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.972193 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa3e55fc83243a3b4978e981fdadf8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:02.973992 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69a8a32a033547b28e591ec3684a668e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:02.978375 890 tablet_service.cc:1467] Processing CreateTablet for tablet ddc83d535408457fa9493592f527f8a4 (DEFAULT_TABLE table=default.Table_809a4f599c56477ca32143d77d2788fc [id=2b9372135fbf4f9cae39a0678260a9f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.978792 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc83d535408457fa9493592f527f8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.979601 1165 ts_tablet_manager.cc:1428] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250109 20:53:02.979934 1165 tablet_bootstrap.cc:492] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:53:02.980211 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1181f2e015334972914ac266d51ac537 consensus state has no leader
I20250109 20:53:02.981441 1165 tablet_bootstrap.cc:654] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.983777 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:02.983932 750 tablet_service.cc:1467] Processing CreateTablet for tablet ddc83d535408457fa9493592f527f8a4 (DEFAULT_TABLE table=default.Table_809a4f599c56477ca32143d77d2788fc [id=2b9372135fbf4f9cae39a0678260a9f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.984226 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc83d535408457fa9493592f527f8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.983541 1158 tablet_bootstrap.cc:654] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:02.985363 747 tablet_service.cc:1467] Processing CreateTablet for tablet f7fa5c14343845b4af4e54b85e3e3064 (DEFAULT_TABLE table=default.Table_a695f0ab86f145438002bbe953aad670 [id=01f191ec5383437880ab609a640c155f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.985455 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:02.985679 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fa5c14343845b4af4e54b85e3e3064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.995767 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9f38f01144874befa5c38a7f6af2d1be
I20250109 20:53:02.997319 883 tablet_service.cc:1467] Processing CreateTablet for tablet bfa3e55fc83243a3b4978e981fdadf8c (DEFAULT_TABLE table=default.Table_908fd543e71a4471a5c0f402e7ac5a7a [id=0b0dd02c61654bf9831b408ca831f2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.997370 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f38f01144874befa5c38a7f6af2d1be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:02.997771 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa3e55fc83243a3b4978e981fdadf8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.997742 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f38f01144874befa5c38a7f6af2d1be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:02.998546 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f7fa5c14343845b4af4e54b85e3e3064 (DEFAULT_TABLE table=default.Table_a695f0ab86f145438002bbe953aad670 [id=01f191ec5383437880ab609a640c155f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.998849 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f38f01144874befa5c38a7f6af2d1be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:02.998957 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fa5c14343845b4af4e54b85e3e3064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.998970 882 tablet_service.cc:1467] Processing CreateTablet for tablet 4136f6e0fc1b46778cd3ab7a3c4dd979 (DEFAULT_TABLE table=default.Table_7aba70359c5c46a9983c0d6750e95591 [id=3a76e4594f5d458d8d86d8729d6a0d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:02.999246 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4136f6e0fc1b46778cd3ab7a3c4dd979. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:02.999788 1032 tablet_service.cc:1467] Processing CreateTablet for tablet ddc83d535408457fa9493592f527f8a4 (DEFAULT_TABLE table=default.Table_809a4f599c56477ca32143d77d2788fc [id=2b9372135fbf4f9cae39a0678260a9f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.000128 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc83d535408457fa9493592f527f8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.002033 1165 tablet_bootstrap.cc:492] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.002205 1165 ts_tablet_manager.cc:1397] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.002s
I20250109 20:53:03.002882 1165 raft_consensus.cc:357] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.003116 1165 raft_consensus.cc:383] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.003168 2368 raft_consensus.cc:491] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.003207 1165 raft_consensus.cc:738] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.003271 2368 raft_consensus.cc:513] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.003398 1165 consensus_queue.cc:260] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.003777 2368 leader_election.cc:290] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.004227 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec1d30b67674facb8b92a2c1a408073" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:03.004779 990 leader_election.cc:343] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.005127 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec1d30b67674facb8b92a2c1a408073" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.005349 1165 ts_tablet_manager.cc:1428] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 20:53:03.005594 1165 tablet_bootstrap.cc:492] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:53:03.005915 989 leader_election.cc:343] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.006006 989 leader_election.cc:304] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.006026 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6bcb56e269714b399e7d1fbc2e2c290e" } new_table_name: "default.Table_a822780af690415a8f728aad79d769db" modify_external_catalogs: true
I20250109 20:53:03.006577 2368 raft_consensus.cc:2743] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.007073 1165 tablet_bootstrap.cc:654] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.009044 449 master_service.cc:946] ReplaceTablet: received request to replace tablet e0f47f9b1f8a479a9760526a4873af57 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:03.005731 747 tablet_service.cc:1467] Processing CreateTablet for tablet 4136f6e0fc1b46778cd3ab7a3c4dd979 (DEFAULT_TABLE table=default.Table_7aba70359c5c46a9983c0d6750e95591 [id=3a76e4594f5d458d8d86d8729d6a0d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.012490 895 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f67c4294f8499496db4e82db8c3bf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.017419 1158 tablet_bootstrap.cc:492] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.018483 1022 tablet_service.cc:1467] Processing CreateTablet for tablet e53966daf1ed4a67801d9ab239e17d8e (DEFAULT_TABLE table=default.Table_62fca2bdab804af8ab92b2eb60115336 [id=689bfafd8fd74c8a8ac06f9a8c68984d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.018682 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4136f6e0fc1b46778cd3ab7a3c4dd979. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.018832 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53966daf1ed4a67801d9ab239e17d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.020480 895 tablet_service.cc:1467] Processing CreateTablet for tablet dd7850715054431288ec6f4350657bc0 (DEFAULT_TABLE table=default.Table_b1b59379d39a4c639dcc4611e0f60767 [id=ffa6c06e279146039d5965344c1a4d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.020897 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7850715054431288ec6f4350657bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.025213 881 tablet_service.cc:1467] Processing CreateTablet for tablet f7fa5c14343845b4af4e54b85e3e3064 (DEFAULT_TABLE table=default.Table_a695f0ab86f145438002bbe953aad670 [id=01f191ec5383437880ab609a640c155f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.025586 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fa5c14343845b4af4e54b85e3e3064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.027792 1158 ts_tablet_manager.cc:1397] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20250109 20:53:03.028461 1157 ts_tablet_manager.cc:1918] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.030107 1176 tablet_bootstrap.cc:492] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.030275 1176 ts_tablet_manager.cc:1397] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.088s user 0.001s sys 0.004s
I20250109 20:53:03.029752 1157 log.cc:1198] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/14fb1017fe6c4f5abd640ad1f6214232
I20250109 20:53:03.031348 1176 raft_consensus.cc:357] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.031579 1176 raft_consensus.cc:383] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.031663 1176 raft_consensus.cc:738] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.031909 1176 consensus_queue.cc:260] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.034507 1165 tablet_bootstrap.cc:492] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.034530 1157 ts_tablet_manager.cc:1939] T 14fb1017fe6c4f5abd640ad1f6214232 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:03.034621 1165 ts_tablet_manager.cc:1397] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.005s
I20250109 20:53:03.035243 1165 raft_consensus.cc:357] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.035398 1165 raft_consensus.cc:383] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.035145 1158 raft_consensus.cc:357] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.035450 1165 raft_consensus.cc:738] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.035591 1158 raft_consensus.cc:383] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.035640 1165 consensus_queue.cc:260] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.035734 1158 raft_consensus.cc:738] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.036136 1165 ts_tablet_manager.cc:1428] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:03.036427 1165 tablet_bootstrap.cc:492] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:03.037909 1165 tablet_bootstrap.cc:654] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.038683 1158 consensus_queue.cc:260] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.040465 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 14fb1017fe6c4f5abd640ad1f6214232 (table default.Table_6c787d10220c41dca63172c9b3bf70ae [id=26365b65a557409abc651578b4458fd4]) successfully deleted
I20250109 20:53:03.046386 1176 ts_tablet_manager.cc:1428] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.016s user 0.000s sys 0.001s
I20250109 20:53:03.050392 1157 tablet_replica.cc:331] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:03.050660 1157 raft_consensus.cc:2238] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.050944 1157 raft_consensus.cc:2267] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.047960 1158 ts_tablet_manager.cc:1428] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250109 20:53:03.052032 1157 ts_tablet_manager.cc:1905] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.052117 1176 tablet_bootstrap.cc:492] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.053905 2368 raft_consensus.cc:491] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.054085 1158 tablet_bootstrap.cc:492] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:03.054018 2368 raft_consensus.cc:513] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.054543 2368 leader_election.cc:290] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.055388 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3af2a00ab04b41915cffc3eb9206bc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:03.055752 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3af2a00ab04b41915cffc3eb9206bc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.056052 1176 tablet_bootstrap.cc:654] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:03.056464 990 leader_election.cc:343] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:03.056684 989 leader_election.cc:343] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.056779 989 leader_election.cc:304] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.057135 2368 raft_consensus.cc:2743] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.059810 750 tablet_service.cc:1467] Processing CreateTablet for tablet e6ab69a3204a4c288b369b5319b1ee0d (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.060182 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ab69a3204a4c288b369b5319b1ee0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.060953 1023 tablet_service.cc:1467] Processing CreateTablet for tablet dd7850715054431288ec6f4350657bc0 (DEFAULT_TABLE table=default.Table_b1b59379d39a4c639dcc4611e0f60767 [id=ffa6c06e279146039d5965344c1a4d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.061527 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7850715054431288ec6f4350657bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.062023 885 tablet_service.cc:1467] Processing CreateTablet for tablet 86f6464cbd12410fa32ed1677c55a3a1 (DEFAULT_TABLE table=default.Table_16735ecbd9074c9ea64459d30539c2f6 [id=01f6377865fc4e9287a8427cd2c92777]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.062337 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f6464cbd12410fa32ed1677c55a3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.063161 756 tablet_service.cc:1467] Processing CreateTablet for tablet a13b6eb4563b46aeb6a89faae823f62a (DEFAULT_TABLE table=default.Table_08105d5a57484ec6b34290b8daa1e416 [id=28cadf9f4ff64ee59c74d752d52f039f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.063409 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13b6eb4563b46aeb6a89faae823f62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.064152 745 tablet_service.cc:1467] Processing CreateTablet for tablet 798aa6062f3447419dbaff60ec445b64 (DEFAULT_TABLE table=default.Table_1d320aee9b99411eb394d0d28de07ce4 [id=ee060b3acde247caaa2cac37bf314a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.064450 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798aa6062f3447419dbaff60ec445b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.065522 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0192f16726024e458eb0eed5b1b5dfd2" } new_table_name: "default.Table_588fb629147c47a7abd46370b0537984" modify_external_catalogs: true
I20250109 20:53:03.066246 1033 tablet_service.cc:1467] Processing CreateTablet for tablet a13b6eb4563b46aeb6a89faae823f62a (DEFAULT_TABLE table=default.Table_08105d5a57484ec6b34290b8daa1e416 [id=28cadf9f4ff64ee59c74d752d52f039f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.066370 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 86f6464cbd12410fa32ed1677c55a3a1 (DEFAULT_TABLE table=default.Table_16735ecbd9074c9ea64459d30539c2f6 [id=01f6377865fc4e9287a8427cd2c92777]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.066552 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13b6eb4563b46aeb6a89faae823f62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.066679 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 6569b450bb7a4a78922061d2ede981fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.066963 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f6464cbd12410fa32ed1677c55a3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.068440 746 tablet_service.cc:1467] Processing CreateTablet for tablet 86f6464cbd12410fa32ed1677c55a3a1 (DEFAULT_TABLE table=default.Table_16735ecbd9074c9ea64459d30539c2f6 [id=01f6377865fc4e9287a8427cd2c92777]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.068744 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f6464cbd12410fa32ed1677c55a3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.070678 916 raft_consensus.cc:3054] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.060981 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 4136f6e0fc1b46778cd3ab7a3c4dd979 (DEFAULT_TABLE table=default.Table_7aba70359c5c46a9983c0d6750e95591 [id=3a76e4594f5d458d8d86d8729d6a0d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.073181 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4136f6e0fc1b46778cd3ab7a3c4dd979. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.077796 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f1c5a6b9681d4ef6a36e308ab2916423 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.077867 893 tablet_service.cc:1467] Processing CreateTablet for tablet 798aa6062f3447419dbaff60ec445b64 (DEFAULT_TABLE table=default.Table_1d320aee9b99411eb394d0d28de07ce4 [id=ee060b3acde247caaa2cac37bf314a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.078186 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798aa6062f3447419dbaff60ec445b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.081712 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 798aa6062f3447419dbaff60ec445b64 (DEFAULT_TABLE table=default.Table_1d320aee9b99411eb394d0d28de07ce4 [id=ee060b3acde247caaa2cac37bf314a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.081996 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798aa6062f3447419dbaff60ec445b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.085091 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 165f6304acc849569a8e90d135a8e2d0 (table default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320]). Expected version 1 got 0
I20250109 20:53:03.087208 747 tablet_service.cc:1467] Processing CreateTablet for tablet dd7850715054431288ec6f4350657bc0 (DEFAULT_TABLE table=default.Table_b1b59379d39a4c639dcc4611e0f60767 [id=ffa6c06e279146039d5965344c1a4d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.088512 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7850715054431288ec6f4350657bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.088685 1165 tablet_bootstrap.cc:492] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.088845 1165 ts_tablet_manager.cc:1397] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.004s
I20250109 20:53:03.089562 1165 raft_consensus.cc:357] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.089735 1165 raft_consensus.cc:383] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.089789 1165 raft_consensus.cc:738] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.090009 1165 consensus_queue.cc:260] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.090569 1165 ts_tablet_manager.cc:1428] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:03.090785 1165 tablet_bootstrap.cc:492] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:03.093137 1165 tablet_bootstrap.cc:654] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.093333 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 17940f311b0440a59a42cc57f3a0bc4c
I20250109 20:53:03.093986 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 17940f311b0440a59a42cc57f3a0bc4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.094058 2368 raft_consensus.cc:491] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.094172 2368 raft_consensus.cc:513] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.057019 887 tablet_service.cc:1467] Processing CreateTablet for tablet 009b90274d6b4830bf7402467f1672df (DEFAULT_TABLE table=default.Table_baf2e0ecb9594bc3af2f86ba420846d4 [id=8ebdd3108bfd4995905f2e90c646d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.095513 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 17940f311b0440a59a42cc57f3a0bc4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.095579 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009b90274d6b4830bf7402467f1672df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.096105 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74a4c1bdd8f4f6e9592ee5d97c58cc6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:03.096614 990 leader_election.cc:343] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.097091 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74a4c1bdd8f4f6e9592ee5d97c58cc6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.097133 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc68008840d4b8fae5185718ab454fa (DEFAULT_TABLE table=default.Table_0c702c1df34a466da3d2c24f0150e438 [id=29268647fb064459a12b1d969b9e2c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.097133 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc68008840d4b8fae5185718ab454fa (DEFAULT_TABLE table=default.Table_0c702c1df34a466da3d2c24f0150e438 [id=29268647fb064459a12b1d969b9e2c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.094698 2368 leader_election.cc:290] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.097405 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc68008840d4b8fae5185718ab454fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.097405 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc68008840d4b8fae5185718ab454fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:03.097705 989 leader_election.cc:343] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.097797 989 leader_election.cc:304] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.098213 2368 raft_consensus.cc:2743] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.099318 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 17940f311b0440a59a42cc57f3a0bc4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.099797 2254 raft_consensus.cc:491] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.099924 2254 raft_consensus.cc:513] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.100417 2254 leader_election.cc:290] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.101061 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cc0bbbb5584f14ba4d6c62211a12c5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.101569 712 leader_election.cc:343] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:03.102239 713 leader_election.cc:343] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 72cc0bbbb5584f14ba4d6c62211a12c5
I20250109 20:53:03.102352 713 leader_election.cc:304] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.102805 2254 raft_consensus.cc:2743] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.103343 1020 tablet_service.cc:1467] Processing CreateTablet for tablet e6ab69a3204a4c288b369b5319b1ee0d (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:03.106114 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0a2efcc2a6ec4b109845d8087491ca9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2efcc2a6ec4b109845d8087491ca9f already in progress: creating tablet
I20250109 20:53:03.106851 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ab69a3204a4c288b369b5319b1ee0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.109201 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc68008840d4b8fae5185718ab454fa (DEFAULT_TABLE table=default.Table_0c702c1df34a466da3d2c24f0150e438 [id=29268647fb064459a12b1d969b9e2c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.110095 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc68008840d4b8fae5185718ab454fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.114176 888 tablet_service.cc:1467] Processing CreateTablet for tablet e6ab69a3204a4c288b369b5319b1ee0d (DEFAULT_TABLE table=default.Table_51b311b735684e74a05c0cd8973a089e [id=1b732f5cb5904387b841b5be096d09c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.114873 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ab69a3204a4c288b369b5319b1ee0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.119649 2032 raft_consensus.cc:491] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.119757 2032 raft_consensus.cc:513] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.120316 2032 leader_election.cc:290] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.120879 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce64e0f3a004c9387bb592cf19236e7" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:03.121785 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce64e0f3a004c9387bb592cf19236e7" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:03.122649 854 leader_election.cc:304] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.123890 2032 raft_consensus.cc:2743] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:03.125221 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdb0ebfad2de4fbf89a4b53a0a150906 consensus state has no leader
I20250109 20:53:03.106439 1166 tablet_replica.cc:331] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:03.129122 1166 raft_consensus.cc:2238] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.130450 1166 raft_consensus.cc:2267] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.134804 1166 ts_tablet_manager.cc:1905] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.139379 1158 tablet_bootstrap.cc:654] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.142719 1176 tablet_bootstrap.cc:492] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.142853 1176 ts_tablet_manager.cc:1397] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.000s
I20250109 20:53:03.143913 1176 raft_consensus.cc:357] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.144299 1176 raft_consensus.cc:383] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.144388 1176 raft_consensus.cc:738] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.144616 1176 consensus_queue.cc:260] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.144766 890 tablet_service.cc:1467] Processing CreateTablet for tablet a13b6eb4563b46aeb6a89faae823f62a (DEFAULT_TABLE table=default.Table_08105d5a57484ec6b34290b8daa1e416 [id=28cadf9f4ff64ee59c74d752d52f039f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.151583 1176 ts_tablet_manager.cc:1428] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:53:03.151898 1176 tablet_bootstrap.cc:492] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.156502 2368 raft_consensus.cc:491] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.156612 2368 raft_consensus.cc:513] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.156682 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.157158 1176 tablet_bootstrap.cc:654] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.157999 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a9aba1c7b424db21c30b2e92323c9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:03.158578 990 leader_election.cc:343] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.157480 2368 leader_election.cc:290] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.159118 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1e89724f79a84156954c1455cfa74676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.159591 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e89724f79a84156954c1455cfa74676 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.160583 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7b2ba41100634eabbefe5b301dc507bd" } modify_external_catalogs: true
I20250109 20:53:03.160739 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7b2ba41100634eabbefe5b301dc507bd" } modify_external_catalogs: true
I20250109 20:53:03.161624 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_27b5a1135e3b433fb92b1f482abeac58" } modify_external_catalogs: true
I20250109 20:53:03.161731 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_27b5a1135e3b433fb92b1f482abeac58" } modify_external_catalogs: true
I20250109 20:53:03.162760 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c720f45ee7754dc3ac46aaa4cbf96871" } new_table_name: "default.Table_cec5108e7ee640ab995d22c9f12d0bae" modify_external_catalogs: true
I20250109 20:53:03.163023 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a9aba1c7b424db21c30b2e92323c9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.163707 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_50d70e8a583c47059eafc658d73feb69" } modify_external_catalogs: true
I20250109 20:53:03.163808 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_50d70e8a583c47059eafc658d73feb69" } modify_external_catalogs: true
I20250109 20:53:03.168342 2254 raft_consensus.cc:491] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.168462 2254 raft_consensus.cc:513] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.169333 2254 leader_election.cc:290] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:03.169552 989 leader_election.cc:343] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.169660 989 leader_election.cc:304] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.169989 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845bac2539e478083bf7fe6cbc0c717" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.170261 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845bac2539e478083bf7fe6cbc0c717" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:03.170433 1053 raft_consensus.cc:2463] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
W20250109 20:53:03.170410 712 leader_election.cc:343] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.170720 713 leader_election.cc:304] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.172375 2368 raft_consensus.cc:2743] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.172610 2254 raft_consensus.cc:2798] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:03.172674 2254 raft_consensus.cc:491] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:03.172734 2254 raft_consensus.cc:3054] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.175454 2254 raft_consensus.cc:513] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.177214 2254 leader_election.cc:290] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.177608 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845bac2539e478083bf7fe6cbc0c717" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:03.177750 1053 raft_consensus.cc:3054] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.178164 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d845bac2539e478083bf7fe6cbc0c717" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:03.178591 712 leader_election.cc:343] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:03.178634 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 152881us
I20250109 20:53:03.180054 1053 raft_consensus.cc:2463] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:03.185173 713 leader_election.cc:304] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.185537 1984 raft_consensus.cc:2798] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:03.185634 1984 raft_consensus.cc:695] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:03.185858 1984 consensus_queue.cc:237] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.186050 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 2b1891cbef8f4b0cae0c15d478db4860 (table default.Table_ca51464e265c4e28b9e54fe7be0587fc [id=0a0599668d77405ba0e72eda2aa5219e]). Expected version 4 got 0
I20250109 20:53:03.188632 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_56a50e008101433aba64aa57711f3688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.189487 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56a50e008101433aba64aa57711f3688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.191385 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d4c1d0cf8d7f48bfbe31ec25d79cd5d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.192023 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4c1d0cf8d7f48bfbe31ec25d79cd5d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.193377 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.197674 2032 raft_consensus.cc:491] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.197799 2032 raft_consensus.cc:513] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:03.198093 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c918f5432574dc38d28609c918131fb consensus state has no leader
I20250109 20:53:03.198355 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 06e615ce8e124cfe9432778b6b58e117 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:03.199575 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13b6eb4563b46aeb6a89faae823f62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.146939 1160 ts_tablet_manager.cc:1918] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.201639 1160 log.cc:1198] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/59e1e8754625463ea42a92ef65a89214
I20250109 20:53:03.201982 1160 ts_tablet_manager.cc:1939] T 59e1e8754625463ea42a92ef65a89214 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:03.203073 1158 tablet_bootstrap.cc:492] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.203253 1158 ts_tablet_manager.cc:1397] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.150s user 0.004s sys 0.000s
I20250109 20:53:03.203940 2032 leader_election.cc:290] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.203999 1158 raft_consensus.cc:357] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.204173 1158 raft_consensus.cc:383] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.204226 1158 raft_consensus.cc:738] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.204445 1158 consensus_queue.cc:260] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.204905 1158 ts_tablet_manager.cc:1428] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:03.205169 1158 tablet_bootstrap.cc:492] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:03.205226 854 leader_election.cc:343] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 3d8d67b624984a0bb327d045c55476da
W20250109 20:53:03.205948 854 leader_election.cc:343] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 3d8d67b624984a0bb327d045c55476da
I20250109 20:53:03.206055 854 leader_election.cc:304] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.206404 2032 raft_consensus.cc:2743] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.206758 1158 tablet_bootstrap.cc:654] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.206750 2409 tablet.cc:1721] T 2b1891cbef8f4b0cae0c15d478db4860 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:03.209933 1165 tablet_bootstrap.cc:492] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.210054 1165 ts_tablet_manager.cc:1397] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.119s user 0.018s sys 0.013s
I20250109 20:53:03.210633 1165 raft_consensus.cc:357] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.210808 1165 raft_consensus.cc:383] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.210858 1165 raft_consensus.cc:738] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.211040 1165 consensus_queue.cc:260] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.211498 1165 ts_tablet_manager.cc:1428] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:03.212607 1165 tablet_bootstrap.cc:492] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:03.213858 1165 tablet_bootstrap.cc:654] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.219977 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 59e1e8754625463ea42a92ef65a89214 (table default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]) successfully deleted
W20250109 20:53:03.224277 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0ac6f8baab5040abbe36e185a8052c6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ac6f8baab5040abbe36e185a8052c6e already in progress: creating tablet
I20250109 20:53:03.226889 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e0f47f9b1f8a479a9760526a4873af57
I20250109 20:53:03.227397 449 catalog_manager.cc:6596] ReplaceTablet: tablet e0f47f9b1f8a479a9760526a4873af57 deleted and replaced by tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7
I20250109 20:53:03.228855 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 2b1891cbef8f4b0cae0c15d478db4860 (table default.Table_ca51464e265c4e28b9e54fe7be0587fc [id=0a0599668d77405ba0e72eda2aa5219e]). Expected version 4 got 0
I20250109 20:53:03.230032 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4506c544ae5f4b2e8b2883e0e5105ddc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.230610 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4506c544ae5f4b2e8b2883e0e5105ddc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:03.231927 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0e0f4e3a82a437ebbb16426fffa5e2f consensus state has no leader
I20250109 20:53:03.232285 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d74b3ee84e164e75a624c43034693fc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.232820 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d74b3ee84e164e75a624c43034693fc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.234961 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 219a4b508050490cb3f1ee418e0edbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.235962 883 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f47f9b1f8a479a9760526a4873af57 with delete_type TABLET_DATA_DELETED (replaced by tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.236215 1158 tablet_bootstrap.cc:492] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.236316 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f47f9b1f8a479a9760526a4873af57 with delete_type TABLET_DATA_DELETED (replaced by tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.236445 1158 ts_tablet_manager.cc:1397] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.004s
I20250109 20:53:03.237713 1158 raft_consensus.cc:357] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.238161 1160 tablet_replica.cc:331] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:03.237985 742 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f47f9b1f8a479a9760526a4873af57 with delete_type TABLET_DATA_DELETED (replaced by tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.238298 1158 raft_consensus.cc:383] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.238361 1158 raft_consensus.cc:738] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.238394 1160 raft_consensus.cc:2238] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.238541 1160 raft_consensus.cc:2267] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.238508 1158 consensus_queue.cc:260] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:03.241219 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 52995.3us
I20250109 20:53:03.246474 1158 ts_tablet_manager.cc:1428] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250109 20:53:03.248958 1158 tablet_bootstrap.cc:492] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:03.250514 1158 tablet_bootstrap.cc:654] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.251255 1165 tablet_bootstrap.cc:492] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.251417 1165 ts_tablet_manager.cc:1397] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.005s
I20250109 20:53:03.252009 1165 raft_consensus.cc:357] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.252239 1165 raft_consensus.cc:383] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.252311 1165 raft_consensus.cc:738] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.252559 1165 consensus_queue.cc:260] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.253055 1165 ts_tablet_manager.cc:1428] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:03.253306 1165 tablet_bootstrap.cc:492] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:03.254251 446 catalog_manager.cc:5526] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:03.254511 1165 tablet_bootstrap.cc:654] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.254894 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.255112 446 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:53:03.261668 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 9d84d315dd4a4e3288a4e5c94deef2de
I20250109 20:53:03.263394 1157 ts_tablet_manager.cc:1918] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.263607 1157 log.cc:1198] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/e16a1d9a5b374b8dbf725eb32ffadb9e
I20250109 20:53:03.265605 1160 ts_tablet_manager.cc:1905] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.268214 1176 tablet_bootstrap.cc:492] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.268419 1176 ts_tablet_manager.cc:1397] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.117s user 0.009s sys 0.000s
I20250109 20:53:03.268788 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8b869f8172ed4fcf815c58fc296752c6" } new_table_name: "default.Table_ec489233361c4a81bb3bdc61530eb072" modify_external_catalogs: true
I20250109 20:53:03.269279 1176 raft_consensus.cc:357] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.269443 1176 raft_consensus.cc:383] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.269490 1176 raft_consensus.cc:738] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.269691 1176 consensus_queue.cc:260] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.270421 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca83a49523b45be9ac7556b40746a83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.276587 1158 tablet_bootstrap.cc:492] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.276785 1158 ts_tablet_manager.cc:1397] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20250109 20:53:03.277426 2368 raft_consensus.cc:491] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.277535 2368 raft_consensus.cc:513] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.277521 1158 raft_consensus.cc:357] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.277673 1158 raft_consensus.cc:383] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.277715 1158 raft_consensus.cc:738] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.277885 1158 consensus_queue.cc:260] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.278077 2368 leader_election.cc:290] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.278664 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165f6304acc849569a8e90d135a8e2d0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.279577 989 leader_election.cc:343] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.279799 2368 raft_consensus.cc:491] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.279906 2368 raft_consensus.cc:513] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.280359 2368 leader_election.cc:290] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.281177 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7587432d7345e4a225fc7d5c968c27" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.281647 989 leader_election.cc:343] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.281832 2368 raft_consensus.cc:491] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.281922 2368 raft_consensus.cc:513] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.282356 2368 leader_election.cc:290] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.282800 2032 raft_consensus.cc:491] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.282971 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8419d8cc7e475ea43dd263cae87aca" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.282953 2032 raft_consensus.cc:513] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:03.283355 989 leader_election.cc:343] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.283634 2032 leader_election.cc:290] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:03.284974 854 leader_election.cc:343] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: cd596614521a4a92a71b781b162f70b6
W20250109 20:53:03.287417 854 leader_election.cc:343] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: cd596614521a4a92a71b781b162f70b6
I20250109 20:53:03.287508 854 leader_election.cc:304] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.287899 2032 raft_consensus.cc:2743] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.282507 1157 ts_tablet_manager.cc:1939] T e16a1d9a5b374b8dbf725eb32ffadb9e P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:03.283219 1158 ts_tablet_manager.cc:1428] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250109 20:53:03.284139 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7587432d7345e4a225fc7d5c968c27" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:03.284936 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8419d8cc7e475ea43dd263cae87aca" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:03.289597 774 raft_consensus.cc:2463] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:53:03.283718 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165f6304acc849569a8e90d135a8e2d0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:03.289745 1176 ts_tablet_manager.cc:1428] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20250109 20:53:03.289765 775 raft_consensus.cc:2463] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
W20250109 20:53:03.290035 990 leader_election.cc:343] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.290115 990 leader_election.cc:304] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.290529 990 leader_election.cc:304] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.290863 2368 raft_consensus.cc:2743] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.291042 2368 raft_consensus.cc:2798] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:03.291100 2368 raft_consensus.cc:491] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:03.291160 2368 raft_consensus.cc:3054] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.291152 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e80959e8c0944167b4ddebf4ebb382c5
I20250109 20:53:03.291931 1158 tablet_bootstrap.cc:492] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:03.292219 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 52b555c0449e4ed280c9116b151b3b7a consensus state has no leader
I20250109 20:53:03.292665 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet be44ffc9e371452f8d5f4174fbfa752a
I20250109 20:53:03.293408 1158 tablet_bootstrap.cc:654] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.293454 2368 raft_consensus.cc:513] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.293689 883 tablet_service.cc:1514] Processing DeleteTablet for tablet be44ffc9e371452f8d5f4174fbfa752a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.293967 2368 leader_election.cc:290] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.294742 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e80959e8c0944167b4ddebf4ebb382c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.294898 990 leader_election.cc:304] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.296191 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8419d8cc7e475ea43dd263cae87aca" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:03.296366 775 raft_consensus.cc:3054] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.296831 2424 raft_consensus.cc:2798] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:03.296730 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8419d8cc7e475ea43dd263cae87aca" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:03.296916 2424 raft_consensus.cc:491] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:03.296964 2424 raft_consensus.cc:3054] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:03.297314 989 leader_election.cc:343] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.298738 775 raft_consensus.cc:2463] T 4f8419d8cc7e475ea43dd263cae87aca P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:03.299121 2424 raft_consensus.cc:513] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.301632 1158 tablet_bootstrap.cc:492] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.304353 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet e16a1d9a5b374b8dbf725eb32ffadb9e (table default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]) successfully deleted
I20250109 20:53:03.301739 742 tablet_service.cc:1514] Processing DeleteTablet for tablet e80959e8c0944167b4ddebf4ebb382c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.301985 746 tablet_service.cc:1514] Processing DeleteTablet for tablet be44ffc9e371452f8d5f4174fbfa752a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.305230 1158 ts_tablet_manager.cc:1397] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20250109 20:53:03.306175 1158 raft_consensus.cc:357] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.299193 990 leader_election.cc:304] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.302124 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet be44ffc9e371452f8d5f4174fbfa752a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.307340 2368 raft_consensus.cc:2798] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:03.308367 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165f6304acc849569a8e90d135a8e2d0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:03.302682 1176 tablet_bootstrap.cc:492] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.308549 775 raft_consensus.cc:3054] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.308951 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165f6304acc849569a8e90d135a8e2d0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:03.309394 989 leader_election.cc:343] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.308387 1158 raft_consensus.cc:383] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.309597 1158 raft_consensus.cc:738] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.309795 1158 consensus_queue.cc:260] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.310322 1158 ts_tablet_manager.cc:1428] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20250109 20:53:03.311610 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e80959e8c0944167b4ddebf4ebb382c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.311667 775 raft_consensus.cc:2463] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:03.312155 990 leader_election.cc:304] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.312390 1176 tablet_bootstrap.cc:654] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.313562 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3074f0a98179449ab995c6544812e329"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.314049 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3074f0a98179449ab995c6544812e329 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.316329 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3254b432932b4d9bb83cb8a2f26e5514" } modify_external_catalogs: true
I20250109 20:53:03.316488 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3254b432932b4d9bb83cb8a2f26e5514" } modify_external_catalogs: true
I20250109 20:53:03.316808 2424 leader_election.cc:290] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.317696 2424 raft_consensus.cc:2798] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:03.317983 2424 raft_consensus.cc:695] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:53:03.319172 1158 tablet_bootstrap.cc:492] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:03.324682 1158 tablet_bootstrap.cc:654] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.325294 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2c68d926e1b8418aa0339da1c4cb17cf" } modify_external_catalogs: true
I20250109 20:53:03.325443 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2c68d926e1b8418aa0339da1c4cb17cf" } modify_external_catalogs: true
I20250109 20:53:03.326216 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_80c12feb8e5f49aca8a5d784a324e389"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.326751 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80c12feb8e5f49aca8a5d784a324e389 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.332229 1166 ts_tablet_manager.cc:1918] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.332336 1166 log.cc:1198] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/712acab92e0f44a3a585d7d871498aa3
I20250109 20:53:03.332618 1166 ts_tablet_manager.cc:1939] T 712acab92e0f44a3a585d7d871498aa3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:03.334026 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 06e615ce8e124cfe9432778b6b58e117
I20250109 20:53:03.334388 454 catalog_manager.cc:6596] ReplaceTablet: tablet 06e615ce8e124cfe9432778b6b58e117 deleted and replaced by tablet e7e7f4dde1d641028863e5a9692bf282
I20250109 20:53:03.334642 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_70499a7567fa4d6498705309cde3b257" } new_table_name: "default.Table_029561426fdb42bba6f6ec539f655c01" modify_external_catalogs: true
I20250109 20:53:03.335906 1157 tablet_replica.cc:331] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:03.336016 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e615ce8e124cfe9432778b6b58e117 with delete_type TABLET_DATA_DELETED (replaced by tablet e7e7f4dde1d641028863e5a9692bf282) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.336097 1157 raft_consensus.cc:2238] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.336267 1157 raft_consensus.cc:2267] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.336584 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e615ce8e124cfe9432778b6b58e117 with delete_type TABLET_DATA_DELETED (replaced by tablet e7e7f4dde1d641028863e5a9692bf282) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.336861 1157 ts_tablet_manager.cc:1905] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.346926 1160 ts_tablet_manager.cc:1918] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:53:03.351260 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 20d27588609f465c8dedae173e57e9b2 (table default.Table_d2938bca74b84f31a863e8a06dd3fd6b [id=2c8afc281d7c46cab5824b64446e9403], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 20d27588609f465c8dedae173e57e9b2 consensus state has no leader
I20250109 20:53:03.351552 1160 log.cc:1198] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/aa7d7695ff5946abb6ee576431d2e87f
I20250109 20:53:03.352372 1160 ts_tablet_manager.cc:1939] T aa7d7695ff5946abb6ee576431d2e87f P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:03.353439 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet aa7d7695ff5946abb6ee576431d2e87f (table default.Table_d311b482665944949ba9d26a91d7fd39 [id=d1eb369ad1da4221b0be6419fe55ad33]) successfully deleted
W20250109 20:53:03.358722 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb57395c2e684e878baf53d76802dc45 consensus state has no leader
I20250109 20:53:03.359202 919 raft_consensus.cc:3054] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:03.364915 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ab7d4f31211742e3b23e9f22e36cf7c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab7d4f31211742e3b23e9f22e36cf7c7 already in progress: creating tablet
I20250109 20:53:03.366313 2032 raft_consensus.cc:491] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.366446 2032 raft_consensus.cc:513] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.366850 1160 tablet_replica.cc:331] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:03.367007 2032 leader_election.cc:290] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:03.367846 854 leader_election.cc:343] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: f1c5a6b9681d4ef6a36e308ab2916423
I20250109 20:53:03.368198 2417 raft_consensus.cc:491] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.368247 1160 raft_consensus.cc:2238] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.368387 1160 raft_consensus.cc:2267] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.368337 2417 raft_consensus.cc:513] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.368916 1160 ts_tablet_manager.cc:1905] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.369092 2417 leader_election.cc:290] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.369797 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.370386 712 leader_election.cc:343] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.373935 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e615ce8e124cfe9432778b6b58e117 with delete_type TABLET_DATA_DELETED (replaced by tablet e7e7f4dde1d641028863e5a9692bf282) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.385162 1165 tablet_bootstrap.cc:492] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.318399 2424 consensus_queue.cc:237] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.391240 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1138) took 1166 ms. Trace:
I20250109 20:53:03.391414 1166 rpcz_store.cc:276] 0109 20:53:02.224307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.224369 (+ 62us) service_pool.cc:225] Handling call
0109 20:53:03.106396 (+882027us) ts_tablet_manager.cc:1145] Deleting tablet 712acab92e0f44a3a585d7d871498aa3
0109 20:53:03.106405 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.191366 (+ 84961us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.332127 (+140761us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.391210 (+ 59083us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":881838}
I20250109 20:53:03.392159 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 712acab92e0f44a3a585d7d871498aa3 (table default.Table_1f4279f07e8b4e0a86928bdb9a67bbf6 [id=8bec14b8b4b74d9c87fe86950f5128e1]) successfully deleted
I20250109 20:53:03.392261 1166 tablet_replica.cc:331] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:03.392642 1166 raft_consensus.cc:2238] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.385330 1165 ts_tablet_manager.cc:1397] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.132s user 0.010s sys 0.015s
I20250109 20:53:03.401144 1165 raft_consensus.cc:357] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.401357 1165 raft_consensus.cc:383] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.401412 1165 raft_consensus.cc:738] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.401600 1165 consensus_queue.cc:260] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.402091 1165 ts_tablet_manager.cc:1428] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250109 20:53:03.406199 854 leader_election.cc:343] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: f1c5a6b9681d4ef6a36e308ab2916423
I20250109 20:53:03.406328 854 leader_election.cc:304] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.407275 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:03.407475 1055 raft_consensus.cc:2463] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:03.407967 713 leader_election.cc:304] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.408010 2032 raft_consensus.cc:2743] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.408439 2417 raft_consensus.cc:2798] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:03.408514 2417 raft_consensus.cc:491] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:03.408599 2417 raft_consensus.cc:3054] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.411300 2417 raft_consensus.cc:513] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.412205 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:03.412730 712 leader_election.cc:343] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.413256 2417 leader_election.cc:290] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.415541 1165 tablet_bootstrap.cc:492] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:03.417444 1165 tablet_bootstrap.cc:654] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.424832 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a0874353369a4986b5b3e814940a97a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.425524 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0874353369a4986b5b3e814940a97a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.427004 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dce6c039ff03443199818ad6d730929d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.427402 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dce6c039ff03443199818ad6d730929d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.429136 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ff364e95ed60447ca22919c7fba6e242"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.429631 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff364e95ed60447ca22919c7fba6e242 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.431885 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7bb113fc7cc443749bc8851afc2fc51e" } new_table_name: "default.Table_c308f00a936a4895b1d9d35b5df4b50a" modify_external_catalogs: true
I20250109 20:53:03.432981 2409 tablet.cc:1721] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:03.433755 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.433804 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.434300 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f85944f66543fe8dd634ad0da0728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.434499 747 tablet_service.cc:1514] Processing DeleteTablet for tablet ea477dc845614ea78e71e45f862b57ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.436450 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:03.436642 1055 raft_consensus.cc:3054] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.439529 1055 raft_consensus.cc:2463] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:03.439915 713 leader_election.cc:304] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.440200 2417 raft_consensus.cc:2798] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:03.440351 2417 raft_consensus.cc:695] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:03.440639 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_08105d5a57484ec6b34290b8daa1e416" } new_table_name: "default.Table_f4204ccf6eb1426192463345ae7109a4" modify_external_catalogs: true
I20250109 20:53:03.440635 2417 consensus_queue.cc:237] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.448964 1165 tablet_bootstrap.cc:492] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.449187 1165 ts_tablet_manager.cc:1397] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.002s
I20250109 20:53:03.449497 2437 tablet.cc:1721] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:03.449920 1165 raft_consensus.cc:357] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.450098 1165 raft_consensus.cc:383] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.450145 1165 raft_consensus.cc:738] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.450814 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_67e322e415e7458f88f98651dd786dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.451316 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67e322e415e7458f88f98651dd786dfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.450511 1165 consensus_queue.cc:260] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.452981 1165 ts_tablet_manager.cc:1428] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:53:03.453670 1165 tablet_bootstrap.cc:492] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:03.455792 1165 tablet_bootstrap.cc:654] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.460567 2368 raft_consensus.cc:695] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:53:03.460913 2368 consensus_queue.cc:237] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:03.461633 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 149486us
I20250109 20:53:03.461910 2428 raft_consensus.cc:491] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.462114 2428 raft_consensus.cc:513] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.463305 2428 leader_election.cc:290] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.463385 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd62a3ae64a143998bb1172b3a97c69f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.463958 989 leader_election.cc:343] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.466434 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd62a3ae64a143998bb1172b3a97c69f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:03.471101 990 leader_election.cc:343] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.471263 990 leader_election.cc:304] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.471513 2428 raft_consensus.cc:2743] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:03.471740 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6b23623677b4e088c14af925b4e07d4 consensus state has no leader
I20250109 20:53:03.472071 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b1e5a40512794ab9ad99471666d8f7b7" } modify_external_catalogs: true
I20250109 20:53:03.472198 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b1e5a40512794ab9ad99471666d8f7b7" } modify_external_catalogs: true
W20250109 20:53:03.472612 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a13b6eb4563b46aeb6a89faae823f62a consensus state has no leader
I20250109 20:53:03.476593 888 tablet_service.cc:1467] Processing CreateTablet for tablet c913dc6da1a0401596634d361d025164 (DEFAULT_TABLE table=default.Table_9014025f02f448f7bad5dd2f9cd1d8d8 [id=d9ecf170aebd47338e10d5a2a7078742]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:03.476722 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7aa8b2c8e744aa3a1ac70fef238e65b consensus state has no leader
I20250109 20:53:03.393011 1166 raft_consensus.cc:2267] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.478758 1158 tablet_bootstrap.cc:492] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.478761 890 tablet_service.cc:1467] Processing CreateTablet for tablet f9494dfd3c414c6ba111bbddb6f4a41b (DEFAULT_TABLE table=default.Table_7e4416ccc3ff49dab092f17ee96df2e9 [id=93c8427c9b5548fdbb2383e9fda75c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.478871 1158 ts_tablet_manager.cc:1397] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.160s user 0.010s sys 0.014s
I20250109 20:53:03.478864 1022 tablet_service.cc:1467] Processing CreateTablet for tablet c913dc6da1a0401596634d361d025164 (DEFAULT_TABLE table=default.Table_9014025f02f448f7bad5dd2f9cd1d8d8 [id=d9ecf170aebd47338e10d5a2a7078742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.479090 1166 ts_tablet_manager.cc:1905] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.479177 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c913dc6da1a0401596634d361d025164. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.479624 1158 raft_consensus.cc:357] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.479815 1158 raft_consensus.cc:383] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.479868 1158 raft_consensus.cc:738] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.480048 1158 consensus_queue.cc:260] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.480731 1158 ts_tablet_manager.cc:1428] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:03.481009 1158 tablet_bootstrap.cc:492] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:03.484629 1158 tablet_bootstrap.cc:654] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.487325 2428 raft_consensus.cc:491] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.487442 2428 raft_consensus.cc:513] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.487643 448 catalog_manager.cc:5526] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:03.487946 2428 leader_election.cc:290] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:03.488034 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]). Expected version 1 got 0
I20250109 20:53:03.489192 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4abc19297e4029b1b7c5396b49b967" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:03.489768 990 leader_election.cc:343] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.491948 747 tablet_service.cc:1467] Processing CreateTablet for tablet f9494dfd3c414c6ba111bbddb6f4a41b (DEFAULT_TABLE table=default.Table_7e4416ccc3ff49dab092f17ee96df2e9 [id=93c8427c9b5548fdbb2383e9fda75c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.492372 742 tablet_service.cc:1467] Processing CreateTablet for tablet c913dc6da1a0401596634d361d025164 (DEFAULT_TABLE table=default.Table_9014025f02f448f7bad5dd2f9cd1d8d8 [id=d9ecf170aebd47338e10d5a2a7078742]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.493381 1020 tablet_service.cc:1467] Processing CreateTablet for tablet f9494dfd3c414c6ba111bbddb6f4a41b (DEFAULT_TABLE table=default.Table_7e4416ccc3ff49dab092f17ee96df2e9 [id=93c8427c9b5548fdbb2383e9fda75c27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.493515 883 tablet_service.cc:1467] Processing CreateTablet for tablet 80b36785104f4bb1a5af3fa2cff6c5ac (DEFAULT_TABLE table=default.Table_1c0dbdff917446539c1bbfa0600d5bbf [id=02370b833f284497bef9ccc3c1b749f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.494076 746 tablet_service.cc:1467] Processing CreateTablet for tablet 80b36785104f4bb1a5af3fa2cff6c5ac (DEFAULT_TABLE table=default.Table_1c0dbdff917446539c1bbfa0600d5bbf [id=02370b833f284497bef9ccc3c1b749f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.494642 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9494dfd3c414c6ba111bbddb6f4a41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.495553 750 tablet_service.cc:1467] Processing CreateTablet for tablet 5215f8d72d47446ba4a713aa90172bbf (DEFAULT_TABLE table=default.Table_c8cea69ded7f4343949b6bd1b8ab7b1e [id=986dd89be7494e51ba6bdc45f3d098d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.495613 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4abc19297e4029b1b7c5396b49b967" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.496093 893 tablet_service.cc:1467] Processing CreateTablet for tablet 5215f8d72d47446ba4a713aa90172bbf (DEFAULT_TABLE table=default.Table_c8cea69ded7f4343949b6bd1b8ab7b1e [id=986dd89be7494e51ba6bdc45f3d098d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:03.496496 989 leader_election.cc:343] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.496587 989 leader_election.cc:304] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.496943 2428 raft_consensus.cc:2743] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.498081 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 5215f8d72d47446ba4a713aa90172bbf (DEFAULT_TABLE table=default.Table_c8cea69ded7f4343949b6bd1b8ab7b1e [id=986dd89be7494e51ba6bdc45f3d098d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.498648 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5215f8d72d47446ba4a713aa90172bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.498615 449 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]). Expected version 3 got 0
I20250109 20:53:03.499516 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_21ed68ee9ed84331ae175e7fe5d6bd6f" } new_table_name: "default.Table_d12ce8f92d214baeb25b460ac371d45e" modify_external_catalogs: true
I20250109 20:53:03.500597 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7ab416cb67944fa5b70140a8b8ab5e45" } modify_external_catalogs: true
I20250109 20:53:03.500742 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7ab416cb67944fa5b70140a8b8ab5e45" } modify_external_catalogs: true
I20250109 20:53:03.502449 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a39fac67894a208dd7ec3153b05e4a with delete_type TABLET_DATA_DELETED (replaced by tablet 9629fe30752f49478afad9b93dc235eb) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.502744 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_14c64344192b45778141de193a1c433d" } modify_external_catalogs: true
I20250109 20:53:03.502894 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_14c64344192b45778141de193a1c433d" } modify_external_catalogs: true
I20250109 20:53:03.502954 2413 raft_consensus.cc:491] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.503088 2413 raft_consensus.cc:513] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.503688 2413 leader_election.cc:290] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.507725 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b36785104f4bb1a5af3fa2cff6c5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.508913 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5215f8d72d47446ba4a713aa90172bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.509402 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c913dc6da1a0401596634d361d025164. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:03.509550 713 leader_election.cc:343] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: ea477dc845614ea78e71e45f862b57ff
I20250109 20:53:03.509809 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9494dfd3c414c6ba111bbddb6f4a41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.509793 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea477dc845614ea78e71e45f862b57ff" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.512995 712 leader_election.cc:343] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.513162 712 leader_election.cc:304] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.513568 2413 raft_consensus.cc:2743] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.518215 1176 tablet_bootstrap.cc:492] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.518422 1176 ts_tablet_manager.cc:1397] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.216s user 0.010s sys 0.005s
I20250109 20:53:03.519204 1176 raft_consensus.cc:357] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.519445 1176 raft_consensus.cc:383] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.519529 1176 raft_consensus.cc:738] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.519762 1176 consensus_queue.cc:260] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.520952 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b36785104f4bb1a5af3fa2cff6c5ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:03.521109 989 consensus_peers.cc:487] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:03.521378 1158 tablet_bootstrap.cc:492] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.521517 1158 ts_tablet_manager.cc:1397] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.002s
I20250109 20:53:03.521781 1176 ts_tablet_manager.cc:1428] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:53:03.522357 1176 tablet_bootstrap.cc:492] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.522377 1158 raft_consensus.cc:357] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.522572 1158 raft_consensus.cc:383] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.522625 1158 raft_consensus.cc:738] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.522831 1158 consensus_queue.cc:260] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.523427 1158 ts_tablet_manager.cc:1428] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:03.523695 1158 tablet_bootstrap.cc:492] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:03.524881 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c913dc6da1a0401596634d361d025164. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.525131 1158 tablet_bootstrap.cc:654] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.530683 452 catalog_manager.cc:5526] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
I20250109 20:53:03.531035 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 165f6304acc849569a8e90d135a8e2d0 (table default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320]). Expected version 1 got 0
I20250109 20:53:03.531237 452 catalog_manager.cc:5526] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
I20250109 20:53:03.532186 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c55a378736c64b6ea978572ddc8c6e4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.532627 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c55a378736c64b6ea978572ddc8c6e4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.533576 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5215f8d72d47446ba4a713aa90172bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.533984 2443 tablet.cc:1721] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:03.536099 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 1e66b7acae984dd6911c3fbf1a06780d (DEFAULT_TABLE table=default.Table_45be879858bb4f5c9c9513bec5a7d2ec [id=d5c54db0d58543fab0154cc3e406e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.536569 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e66b7acae984dd6911c3fbf1a06780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.536979 746 tablet_service.cc:1467] Processing CreateTablet for tablet 456f428f30d440b3b52fd2dcb05d437d (DEFAULT_TABLE table=default.Table_cb96e72193f94850b49ec5ac33b2cbe5 [id=cb9b36403ff14236afd8a262f3ef97b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.538640 887 tablet_service.cc:1467] Processing CreateTablet for tablet 456f428f30d440b3b52fd2dcb05d437d (DEFAULT_TABLE table=default.Table_cb96e72193f94850b49ec5ac33b2cbe5 [id=cb9b36403ff14236afd8a262f3ef97b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.538928 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456f428f30d440b3b52fd2dcb05d437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.542158 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 456f428f30d440b3b52fd2dcb05d437d (DEFAULT_TABLE table=default.Table_cb96e72193f94850b49ec5ac33b2cbe5 [id=cb9b36403ff14236afd8a262f3ef97b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.542461 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456f428f30d440b3b52fd2dcb05d437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.544553 882 tablet_service.cc:1467] Processing CreateTablet for tablet 1e66b7acae984dd6911c3fbf1a06780d (DEFAULT_TABLE table=default.Table_45be879858bb4f5c9c9513bec5a7d2ec [id=d5c54db0d58543fab0154cc3e406e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.544852 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e66b7acae984dd6911c3fbf1a06780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.547955 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9494dfd3c414c6ba111bbddb6f4a41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.552982 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 80b36785104f4bb1a5af3fa2cff6c5ac (DEFAULT_TABLE table=default.Table_1c0dbdff917446539c1bbfa0600d5bbf [id=02370b833f284497bef9ccc3c1b749f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.553258 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b36785104f4bb1a5af3fa2cff6c5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.555886 1160 ts_tablet_manager.cc:1918] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.555984 1160 log.cc:1198] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/a1ed00f7954143c9ba9bf65d8a8a68d2
I20250109 20:53:03.556255 1160 ts_tablet_manager.cc:1939] T a1ed00f7954143c9ba9bf65d8a8a68d2 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:03.559567 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3df9ce1c679145179722f6c8d0436e03 (DEFAULT_TABLE table=default.Table_7dacb59e95464e77b88e68386776f17d [id=a521b026ac614c968fc6982070f7230e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.559875 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df9ce1c679145179722f6c8d0436e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.560681 893 tablet_service.cc:1467] Processing CreateTablet for tablet 5ced7e77dcb4404d95175dab42615cee (DEFAULT_TABLE table=default.Table_86f22049bc704f91b332d2205d149706 [id=af2d4db448f04f5bac9219bd4d3e4a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.560968 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ced7e77dcb4404d95175dab42615cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.561944 745 tablet_service.cc:1467] Processing CreateTablet for tablet 1a14244714a14158bc895344a5776974 (DEFAULT_TABLE table=default.Table_0ad28a15c08542d3bdf5e80b70702d85 [id=76049ff6239b4e7cbee34235ef6a78a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.562250 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a14244714a14158bc895344a5776974. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.563810 1176 tablet_bootstrap.cc:654] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.564771 883 tablet_service.cc:1467] Processing CreateTablet for tablet 1a14244714a14158bc895344a5776974 (DEFAULT_TABLE table=default.Table_0ad28a15c08542d3bdf5e80b70702d85 [id=76049ff6239b4e7cbee34235ef6a78a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.565068 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a14244714a14158bc895344a5776974. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.567567 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456f428f30d440b3b52fd2dcb05d437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.568538 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 5ced7e77dcb4404d95175dab42615cee (DEFAULT_TABLE table=default.Table_86f22049bc704f91b332d2205d149706 [id=af2d4db448f04f5bac9219bd4d3e4a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.569310 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ced7e77dcb4404d95175dab42615cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.572944 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 3df9ce1c679145179722f6c8d0436e03 (DEFAULT_TABLE table=default.Table_7dacb59e95464e77b88e68386776f17d [id=a521b026ac614c968fc6982070f7230e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.573278 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df9ce1c679145179722f6c8d0436e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.575459 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet a1ed00f7954143c9ba9bf65d8a8a68d2 (table default.Table_f739863bb39f41698e7e5a083fe352dd [id=c17cff049d174e1d94c2d55712e886d7]) successfully deleted
W20250109 20:53:03.576429 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 14fb1017fe6c4f5abd640ad1f6214232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14fb1017fe6c4f5abd640ad1f6214232 already in progress: creating tablet
W20250109 20:53:03.579161 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 90bcfd699e094c499df625281e7119b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90bcfd699e094c499df625281e7119b1 already in progress: creating tablet
I20250109 20:53:03.579337 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:03.580235 1160 tablet_replica.cc:331] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:03.580394 1160 raft_consensus.cc:2238] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.580531 1160 raft_consensus.cc:2267] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.582155 1160 ts_tablet_manager.cc:1905] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.568975 742 tablet_service.cc:1467] Processing CreateTablet for tablet 5ced7e77dcb4404d95175dab42615cee (DEFAULT_TABLE table=default.Table_86f22049bc704f91b332d2205d149706 [id=af2d4db448f04f5bac9219bd4d3e4a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.576854 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3df9ce1c679145179722f6c8d0436e03 (DEFAULT_TABLE table=default.Table_7dacb59e95464e77b88e68386776f17d [id=a521b026ac614c968fc6982070f7230e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.584002 750 tablet_service.cc:1467] Processing CreateTablet for tablet 1e66b7acae984dd6911c3fbf1a06780d (DEFAULT_TABLE table=default.Table_45be879858bb4f5c9c9513bec5a7d2ec [id=d5c54db0d58543fab0154cc3e406e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.585992 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e66b7acae984dd6911c3fbf1a06780d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.586902 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 1a14244714a14158bc895344a5776974 (DEFAULT_TABLE table=default.Table_0ad28a15c08542d3bdf5e80b70702d85 [id=76049ff6239b4e7cbee34235ef6a78a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.587208 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a14244714a14158bc895344a5776974. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.588498 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ced7e77dcb4404d95175dab42615cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.595124 883 tablet_service.cc:1467] Processing CreateTablet for tablet 054fc6d1b6154da79f4c537aab34dd3f (DEFAULT_TABLE table=default.Table_90f05f2d38db4882b476617d20ba76ef [id=7700dc8458224237b56dde6134510989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.596676 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df9ce1c679145179722f6c8d0436e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.512670 1166 ts_tablet_manager.cc:1918] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.597648 1166 log.cc:1198] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/e716270ffcf44039b075023fd0c2d2a3
I20250109 20:53:03.597662 888 tablet_service.cc:1467] Processing CreateTablet for tablet e560e5b490774c288157294ad6f23a66 (DEFAULT_TABLE table=default.Table_1359ff2be59e4f6683ac7b4577a85027 [id=442ee55f2e6e45d8918d5016d7d1f873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.598116 1166 ts_tablet_manager.cc:1939] T e716270ffcf44039b075023fd0c2d2a3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:03.598400 887 tablet_service.cc:1467] Processing CreateTablet for tablet f3371f56ddfc4ad08d87caae70ce0722 (DEFAULT_TABLE table=default.Table_4c067bae363a4fdfa0fd5de56862c35c [id=542e2b16a8d6474d89bab60f9d9201c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.597285 893 tablet_service.cc:1467] Processing CreateTablet for tablet 88479a3f57654d84ad622af0782cce47 (DEFAULT_TABLE table=default.Table_43b9440786c14eb0af7ca7e0aadf887b [id=d90a1147b77349458ab0702ad915bd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.598914 2032 raft_consensus.cc:491] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.599012 2032 raft_consensus.cc:513] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.599211 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1139) took 1374 ms. Trace:
I20250109 20:53:03.599323 1166 rpcz_store.cc:276] 0109 20:53:02.224437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.224604 (+ 167us) service_pool.cc:225] Handling call
0109 20:53:03.392157 (+1167553us) ts_tablet_manager.cc:1145] Deleting tablet e716270ffcf44039b075023fd0c2d2a3
0109 20:53:03.392191 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.509872 (+117681us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.512608 (+ 2736us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.599172 (+ 86564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1167430}
I20250109 20:53:03.599517 2032 leader_election.cc:290] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.599920 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1140) took 1179 ms. Trace:
I20250109 20:53:03.600026 1166 rpcz_store.cc:276] 0109 20:53:02.420336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.420388 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:03.599774 (+1179386us) ts_tablet_manager.cc:1145] Deleting tablet d3428128a5494341b1986ce6956bb88a
0109 20:53:03.599783 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.599906 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179198}
I20250109 20:53:03.600361 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet e716270ffcf44039b075023fd0c2d2a3 (table default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]) successfully deleted
W20250109 20:53:03.600374 854 leader_election.cc:343] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 0b0450d88e3e4ccaae4e5157ce5c80c1
I20250109 20:53:03.601150 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 88479a3f57654d84ad622af0782cce47 (DEFAULT_TABLE table=default.Table_43b9440786c14eb0af7ca7e0aadf887b [id=d90a1147b77349458ab0702ad915bd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.601150 1033 tablet_service.cc:1467] Processing CreateTablet for tablet e560e5b490774c288157294ad6f23a66 (DEFAULT_TABLE table=default.Table_1359ff2be59e4f6683ac7b4577a85027 [id=442ee55f2e6e45d8918d5016d7d1f873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.601606 756 tablet_service.cc:1467] Processing CreateTablet for tablet e560e5b490774c288157294ad6f23a66 (DEFAULT_TABLE table=default.Table_1359ff2be59e4f6683ac7b4577a85027 [id=442ee55f2e6e45d8918d5016d7d1f873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.601913 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e560e5b490774c288157294ad6f23a66. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:03.602034 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d3428128a5494341b1986ce6956bb88a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3428128a5494341b1986ce6956bb88a already in progress: creating tablet
I20250109 20:53:03.602391 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 054fc6d1b6154da79f4c537aab34dd3f (DEFAULT_TABLE table=default.Table_90f05f2d38db4882b476617d20ba76ef [id=7700dc8458224237b56dde6134510989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.603926 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1141) took 1159 ms. Trace:
I20250109 20:53:03.604355 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88479a3f57654d84ad622af0782cce47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.604210 1166 rpcz_store.cc:276] 0109 20:53:02.444324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.444616 (+ 292us) service_pool.cc:225] Handling call
0109 20:53:03.600416 (+1155800us) ts_tablet_manager.cc:1145] Deleting tablet ef0f27e5506444fd996d9ded42dfacb7
0109 20:53:03.603526 (+ 3110us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.603812 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155587}
W20250109 20:53:03.605079 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ef0f27e5506444fd996d9ded42dfacb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef0f27e5506444fd996d9ded42dfacb7 already in progress: creating tablet
I20250109 20:53:03.605670 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e560e5b490774c288157294ad6f23a66. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:03.606443 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0ac6f8baab5040abbe36e185a8052c6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ac6f8baab5040abbe36e185a8052c6e already in progress: creating tablet
I20250109 20:53:03.601899 745 tablet_service.cc:1467] Processing CreateTablet for tablet 88479a3f57654d84ad622af0782cce47 (DEFAULT_TABLE table=default.Table_43b9440786c14eb0af7ca7e0aadf887b [id=d90a1147b77349458ab0702ad915bd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.609989 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88479a3f57654d84ad622af0782cce47. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:03.610466 854 leader_election.cc:343] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 0b0450d88e3e4ccaae4e5157ce5c80c1
I20250109 20:53:03.610561 854 leader_election.cc:304] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.610884 2032 raft_consensus.cc:2743] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.617105 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054fc6d1b6154da79f4c537aab34dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.619127 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054fc6d1b6154da79f4c537aab34dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.619359 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3371f56ddfc4ad08d87caae70ce0722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.619901 1176 tablet_bootstrap.cc:492] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.620012 1176 ts_tablet_manager.cc:1397] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.001s
I20250109 20:53:03.620695 1176 raft_consensus.cc:357] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.620965 1176 raft_consensus.cc:383] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.621062 1176 raft_consensus.cc:738] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.621312 1176 consensus_queue.cc:260] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.622160 1176 ts_tablet_manager.cc:1428] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:03.623137 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e560e5b490774c288157294ad6f23a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.623200 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:03.629451 1020 tablet_service.cc:1467] Processing CreateTablet for tablet f3371f56ddfc4ad08d87caae70ce0722 (DEFAULT_TABLE table=default.Table_4c067bae363a4fdfa0fd5de56862c35c [id=542e2b16a8d6474d89bab60f9d9201c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.629813 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3371f56ddfc4ad08d87caae70ce0722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.630599 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88479a3f57654d84ad622af0782cce47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.631659 745 tablet_service.cc:1467] Processing CreateTablet for tablet b54b66ca10114330b3485f67eb076064 (DEFAULT_TABLE table=default.Table_e8e4034056dd4856a0bdb4c83d34b6d9 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.631970 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54b66ca10114330b3485f67eb076064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.633172 1176 tablet_bootstrap.cc:492] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.634207 887 tablet_service.cc:1467] Processing CreateTablet for tablet b54b66ca10114330b3485f67eb076064 (DEFAULT_TABLE table=default.Table_e8e4034056dd4856a0bdb4c83d34b6d9 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.634367 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b54b66ca10114330b3485f67eb076064 (DEFAULT_TABLE table=default.Table_e8e4034056dd4856a0bdb4c83d34b6d9 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.634611 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54b66ca10114330b3485f67eb076064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.634655 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54b66ca10114330b3485f67eb076064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.636233 1166 tablet_replica.cc:331] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:03.636585 1166 raft_consensus.cc:2238] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:03.636996 1166 pending_rounds.cc:62] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Trying to abort 1 pending ops.
I20250109 20:53:03.637094 1166 pending_rounds.cc:69] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523704522997760 op_type: NO_OP noop_request { }
I20250109 20:53:03.637245 1166 raft_consensus.cc:2883] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:53:03.637399 1166 raft_consensus.cc:2267] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:03.637939 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ab7d4f31211742e3b23e9f22e36cf7c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab7d4f31211742e3b23e9f22e36cf7c7 already in progress: creating tablet
W20250109 20:53:03.638206 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2db4e85cd4094cf0b5b699402346b980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2db4e85cd4094cf0b5b699402346b980 already in progress: creating tablet
I20250109 20:53:03.638242 756 tablet_service.cc:1467] Processing CreateTablet for tablet 70581bd3aa674b73ab1b4908fe1c9030 (DEFAULT_TABLE table=default.Table_ebdd59dd13424ecf93c997fa5919477d [id=79febb0a1443466aaabc6d31347d6ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.638546 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70581bd3aa674b73ab1b4908fe1c9030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.660936 1176 tablet_bootstrap.cc:654] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.662355 1166 ts_tablet_manager.cc:1905] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.665530 2032 raft_consensus.cc:491] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.665676 2032 raft_consensus.cc:513] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.666539 2032 leader_election.cc:290] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:03.667392 854 leader_election.cc:343] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: e0f67c4294f8499496db4e82db8c3bf0
W20250109 20:53:03.667562 854 leader_election.cc:343] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: e0f67c4294f8499496db4e82db8c3bf0
I20250109 20:53:03.667632 854 leader_election.cc:304] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.668115 2032 raft_consensus.cc:2743] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.669499 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5f953f43877c4a738ffd8255afbf8275
I20250109 20:53:03.670174 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f953f43877c4a738ffd8255afbf8275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.670248 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_70b2bddd5393467f8ae2ff2ea7d1a79d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.670732 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70b2bddd5393467f8ae2ff2ea7d1a79d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.670730 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f953f43877c4a738ffd8255afbf8275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.671344 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3293f49190684b72a5c41031e621bd36
I20250109 20:53:03.671738 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6
I20250109 20:53:03.671763 447 master_service.cc:946] ReplaceTablet: received request to replace tablet c6e4cfb5a07841828e5206d100b4f835 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:03.671895 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3293f49190684b72a5c41031e621bd36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.672286 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4511b714cc7844378468fd7bd37c0025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.672662 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4511b714cc7844378468fd7bd37c0025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.672873 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.672976 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9c3af2a00ab04b41915cffc3eb9206bc
I20250109 20:53:03.673053 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3293f49190684b72a5c41031e621bd36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.673496 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0446eb38ff3943c098a5979f8bc09fe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:03.673688 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3af2a00ab04b41915cffc3eb9206bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.672688 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d2c8a6555a8e4228a9a80b15c6880c84
I20250109 20:53:03.673877 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:03.673887 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0446eb38ff3943c098a5979f8bc09fe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.674242 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c8a6555a8e4228a9a80b15c6880c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.674307 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a2dba4139826415bae8547cb15dd9863" } new_table_name: "default.Table_d0d17c135e164e9688c77b42de045b4e" modify_external_catalogs: true
I20250109 20:53:03.674587 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c8a6555a8e4228a9a80b15c6880c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:03.674894 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d841529b90145caad4c4fd19f9c9669 consensus state has no leader
I20250109 20:53:03.675156 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1967d8c334354d17877612396cf2ce5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.675565 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1967d8c334354d17877612396cf2ce5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.676658 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cfdca0092aa74d3e805ce6c6132ca08f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.676998 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfdca0092aa74d3e805ce6c6132ca08f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.679958 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3af2a00ab04b41915cffc3eb9206bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.680210 882 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c8a6555a8e4228a9a80b15c6880c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.680395 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f953f43877c4a738ffd8255afbf8275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.682747 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667] alter complete (version 1)
I20250109 20:53:03.683672 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3293f49190684b72a5c41031e621bd36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.684484 883 tablet_service.cc:1514] Processing DeleteTablet for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.686793 1160 ts_tablet_manager.cc:1918] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.687054 1160 log.cc:1198] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/3d8d67b624984a0bb327d045c55476da
I20250109 20:53:03.687497 1160 ts_tablet_manager.cc:1939] T 3d8d67b624984a0bb327d045c55476da P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:03.688235 1176 tablet_bootstrap.cc:492] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.688407 1176 ts_tablet_manager.cc:1397] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.002s
I20250109 20:53:03.689275 1176 raft_consensus.cc:357] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.689499 1176 raft_consensus.cc:383] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.689592 1176 raft_consensus.cc:738] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.689939 1176 consensus_queue.cc:260] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.692510 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_93fecf5906e3423684391032f8b408c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.693022 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93fecf5906e3423684391032f8b408c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.693797 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 3d8d67b624984a0bb327d045c55476da (table default.Table_6f5675a262804b5b98db035a4c98ddcf [id=4ad56e9706ec48a395cb1335ec602d3d]) successfully deleted
I20250109 20:53:03.694103 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_77fce37f901d442a85f1892961ef0325" } new_table_name: "default.Table_b7132e2ae3e746f3b2e20bb20d2c1398" modify_external_catalogs: true
I20250109 20:53:03.695928 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:03.696105 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2db4e85cd4094cf0b5b699402346b980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2db4e85cd4094cf0b5b699402346b980 already in progress: creating tablet
I20250109 20:53:03.696836 1160 tablet_replica.cc:331] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:03.697330 1176 ts_tablet_manager.cc:1428] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250109 20:53:03.700001 1160 raft_consensus.cc:2238] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.700399 1160 raft_consensus.cc:2267] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.700954 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814361a1e854451a0ee7469cc6d447b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.701059 1160 ts_tablet_manager.cc:1905] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.701182 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3af2a00ab04b41915cffc3eb9206bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.703799 756 tablet_service.cc:1514] Processing DeleteTablet for tablet e090f72eb1e046319b825ee5fe126332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.706897 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.707265 1176 tablet_bootstrap.cc:492] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.710445 1176 tablet_bootstrap.cc:654] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.722733 887 tablet_service.cc:1467] Processing CreateTablet for tablet 70581bd3aa674b73ab1b4908fe1c9030 (DEFAULT_TABLE table=default.Table_ebdd59dd13424ecf93c997fa5919477d [id=79febb0a1443466aaabc6d31347d6ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.723637 756 tablet_service.cc:1467] Processing CreateTablet for tablet b96e0a86e1f04cfb8eb47ff188d6d4f2 (DEFAULT_TABLE table=default.Table_f83de64baea4492f9c52490b335fbd34 [id=e54ea48845934e55a5ff992107c170c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.723891 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70581bd3aa674b73ab1b4908fe1c9030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.724006 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96e0a86e1f04cfb8eb47ff188d6d4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.725164 1033 tablet_service.cc:1467] Processing CreateTablet for tablet a45772ab0e6d4316a011be39d5316b99 (DEFAULT_TABLE table=default.Table_f270a526059d4308925d102e914882ed [id=e2db90856a6f4baa8893fa127e65bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.725503 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45772ab0e6d4316a011be39d5316b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.726516 893 tablet_service.cc:1467] Processing CreateTablet for tablet a45772ab0e6d4316a011be39d5316b99 (DEFAULT_TABLE table=default.Table_f270a526059d4308925d102e914882ed [id=e2db90856a6f4baa8893fa127e65bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.726776 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45772ab0e6d4316a011be39d5316b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.723510 883 tablet_service.cc:1467] Processing CreateTablet for tablet b96e0a86e1f04cfb8eb47ff188d6d4f2 (DEFAULT_TABLE table=default.Table_f83de64baea4492f9c52490b335fbd34 [id=e54ea48845934e55a5ff992107c170c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.728941 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96e0a86e1f04cfb8eb47ff188d6d4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.728971 742 tablet_service.cc:1467] Processing CreateTablet for tablet a45772ab0e6d4316a011be39d5316b99 (DEFAULT_TABLE table=default.Table_f270a526059d4308925d102e914882ed [id=e2db90856a6f4baa8893fa127e65bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.729261 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45772ab0e6d4316a011be39d5316b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.731109 890 tablet_service.cc:1467] Processing CreateTablet for tablet 7946e9a4412b4b54b99f41fe831be795 (DEFAULT_TABLE table=default.Table_9b3a6884c07847b090850ad3f39476db [id=80a41eab2114400bbf74151e57b4916f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.731369 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7946e9a4412b4b54b99f41fe831be795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.732353 751 tablet_service.cc:1467] Processing CreateTablet for tablet a8fe6a54d4184b27928ddb02f188a9f9 (DEFAULT_TABLE table=default.Table_615f5ae1a3c84fbd85db081530862f20 [id=d3cbbb12f8ed4614a52af20b815c33d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.732614 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8fe6a54d4184b27928ddb02f188a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.732975 747 tablet_service.cc:1467] Processing CreateTablet for tablet 7946e9a4412b4b54b99f41fe831be795 (DEFAULT_TABLE table=default.Table_9b3a6884c07847b090850ad3f39476db [id=80a41eab2114400bbf74151e57b4916f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.733273 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7946e9a4412b4b54b99f41fe831be795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.736934 882 tablet_service.cc:1467] Processing CreateTablet for tablet a8fe6a54d4184b27928ddb02f188a9f9 (DEFAULT_TABLE table=default.Table_615f5ae1a3c84fbd85db081530862f20 [id=d3cbbb12f8ed4614a52af20b815c33d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.737226 1166 ts_tablet_manager.cc:1918] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:53:03.737396 1166 log.cc:1198] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/14fb1017fe6c4f5abd640ad1f6214232
I20250109 20:53:03.737588 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8fe6a54d4184b27928ddb02f188a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.737920 1166 ts_tablet_manager.cc:1939] T 14fb1017fe6c4f5abd640ad1f6214232 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:03.738054 1030 tablet_service.cc:1467] Processing CreateTablet for tablet b96e0a86e1f04cfb8eb47ff188d6d4f2 (DEFAULT_TABLE table=default.Table_f83de64baea4492f9c52490b335fbd34 [id=e54ea48845934e55a5ff992107c170c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.738529 746 tablet_service.cc:1467] Processing CreateTablet for tablet f3371f56ddfc4ad08d87caae70ce0722 (DEFAULT_TABLE table=default.Table_4c067bae363a4fdfa0fd5de56862c35c [id=542e2b16a8d6474d89bab60f9d9201c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.738860 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3371f56ddfc4ad08d87caae70ce0722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.739241 2413 raft_consensus.cc:491] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.739354 2413 raft_consensus.cc:513] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.739893 2413 leader_election.cc:290] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.739730 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 7946e9a4412b4b54b99f41fe831be795 (DEFAULT_TABLE table=default.Table_9b3a6884c07847b090850ad3f39476db [id=80a41eab2114400bbf74151e57b4916f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:03.741705 713 leader_election.cc:343] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 18f85944f66543fe8dd634ad0da0728d
I20250109 20:53:03.740571 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f85944f66543fe8dd634ad0da0728d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.742237 712 leader_election.cc:343] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.742326 2032 raft_consensus.cc:491] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.742321 712 leader_election.cc:304] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.742412 2032 raft_consensus.cc:513] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.742841 2413 raft_consensus.cc:2743] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.742918 2032 leader_election.cc:290] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.743315 2368 raft_consensus.cc:491] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.743432 2368 raft_consensus.cc:513] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:03.743714 854 leader_election.cc:343] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 219a4b508050490cb3f1ee418e0edbf8
I20250109 20:53:03.743965 2368 leader_election.cc:290] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:53:03.744347 854 leader_election.cc:343] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 219a4b508050490cb3f1ee418e0edbf8
I20250109 20:53:03.744437 854 leader_election.cc:304] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.744421 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6569b450bb7a4a78922061d2ede981fe" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:03.744840 2032 raft_consensus.cc:2743] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:03.744854 990 leader_election.cc:343] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.745039 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6569b450bb7a4a78922061d2ede981fe" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.745548 989 leader_election.cc:343] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.745635 989 leader_election.cc:304] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:03.746170 2368 raft_consensus.cc:2743] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.627143 1157 ts_tablet_manager.cc:1918] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.753017 1157 log.cc:1198] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/825220629faa4d2481e0db9fa226c306
I20250109 20:53:03.753331 1157 ts_tablet_manager.cc:1939] T 825220629faa4d2481e0db9fa226c306 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:03.754206 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 825220629faa4d2481e0db9fa226c306 (table default.Table_4560626d5c8547678952bb1e7da86ad4 [id=f4c5a8b765814c37882dfdeaf55481e2]) successfully deleted
W20250109 20:53:03.754806 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 90bcfd699e094c499df625281e7119b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90bcfd699e094c499df625281e7119b1 already in progress: creating tablet
I20250109 20:53:03.755115 1157 tablet_replica.cc:331] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:03.755347 1157 raft_consensus.cc:2238] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.755523 1157 raft_consensus.cc:2267] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.756091 1157 ts_tablet_manager.cc:1905] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.756975 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96e0a86e1f04cfb8eb47ff188d6d4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.759943 1160 ts_tablet_manager.cc:1918] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.762639 1176 tablet_bootstrap.cc:492] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.762800 1176 ts_tablet_manager.cc:1397] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.004s
I20250109 20:53:03.763592 1176 raft_consensus.cc:357] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.763796 1176 raft_consensus.cc:383] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.763890 1176 raft_consensus.cc:738] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.764066 750 tablet_service.cc:1467] Processing CreateTablet for tablet 054fc6d1b6154da79f4c537aab34dd3f (DEFAULT_TABLE table=default.Table_90f05f2d38db4882b476617d20ba76ef [id=7700dc8458224237b56dde6134510989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.764102 1176 consensus_queue.cc:260] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.764636 1176 ts_tablet_manager.cc:1428] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:03.765202 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054fc6d1b6154da79f4c537aab34dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.770063 1176 tablet_bootstrap.cc:492] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.771381 1176 tablet_bootstrap.cc:654] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.772004 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_db486b062a6a493397542a98a8deafa8" } new_table_name: "default.Table_d09bd36c6dd447228bb30f6fa48217e8" modify_external_catalogs: true
I20250109 20:53:03.772094 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_32c8c1c279bf47f79cabf5c3332d4076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.772529 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32c8c1c279bf47f79cabf5c3332d4076 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.772891 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c6e4cfb5a07841828e5206d100b4f835
W20250109 20:53:03.773140 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e82cda1841c24bb2a14c127b1f40ad26 consensus state has no leader
I20250109 20:53:03.773273 447 catalog_manager.cc:6596] ReplaceTablet: tablet c6e4cfb5a07841828e5206d100b4f835 deleted and replaced by tablet 3d5e72e3aaea4360b7c8f619a831c1f1
I20250109 20:53:03.773398 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a997c69220da46df9489620287286652" } new_table_name: "default.Table_74b6c2fdf45a4ff38b278a2fd9eda27e" modify_external_catalogs: true
I20250109 20:53:03.773488 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9657151670d0422b879b12781fed0eb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.773856 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9657151670d0422b879b12781fed0eb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.775207 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2e1e70e4ec454bb19ce9f47d5272e0e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.775583 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e1e70e4ec454bb19ce9f47d5272e0e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.778787 2413 raft_consensus.cc:491] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.778925 2413 raft_consensus.cc:513] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.779477 2413 leader_election.cc:290] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:03.780548 713 leader_election.cc:343] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 4160bb4ba40347bda48f8fa88f8b6001
I20250109 20:53:03.781176 756 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e4cfb5a07841828e5206d100b4f835 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d5e72e3aaea4360b7c8f619a831c1f1) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.781948 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e4cfb5a07841828e5206d100b4f835 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d5e72e3aaea4360b7c8f619a831c1f1) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.765955 919 raft_consensus.cc:3054] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:03.779794 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4160bb4ba40347bda48f8fa88f8b6001" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.790129 712 leader_election.cc:343] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.790230 712 leader_election.cc:304] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.791679 2413 raft_consensus.cc:2743] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.794731 2409 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:03.799618 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet cda300a7e58a44339c3bdb2e62a9f384 (table default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0 [id=1284f08f96184576bde56fa61834e33f]). Expected version 1 got 0
I20250109 20:53:03.805207 1158 tablet_bootstrap.cc:492] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.805459 1158 ts_tablet_manager.cc:1397] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.282s user 0.030s sys 0.035s
I20250109 20:53:03.806243 1158 raft_consensus.cc:357] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.806499 1158 raft_consensus.cc:383] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.806582 1158 raft_consensus.cc:738] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.806813 1158 consensus_queue.cc:260] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.808458 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 14fb1017fe6c4f5abd640ad1f6214232 (table default.Table_6c787d10220c41dca63172c9b3bf70ae [id=26365b65a557409abc651578b4458fd4]) successfully deleted
I20250109 20:53:03.808979 1158 ts_tablet_manager.cc:1428] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:53:03.809900 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.811345 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7946e9a4412b4b54b99f41fe831be795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.811520 1158 tablet_bootstrap.cc:492] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:03.811822 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 70581bd3aa674b73ab1b4908fe1c9030 (DEFAULT_TABLE table=default.Table_ebdd59dd13424ecf93c997fa5919477d [id=79febb0a1443466aaabc6d31347d6ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.812204 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70581bd3aa674b73ab1b4908fe1c9030. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:03.812654 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4f6196d820f4aecaff2701aaf56faf5 consensus state has no leader
I20250109 20:53:03.812844 1158 tablet_bootstrap.cc:654] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.821099 893 tablet_service.cc:1467] Processing CreateTablet for tablet 622cc5f673694546a594f968621f1011 (DEFAULT_TABLE table=default.Table_cbf320b748bb4162a3ce1ef93588df7b [id=5ee3e2262ce0468bb5a53579df41168b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.821146 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 413341019ff94a10896d956f5a03c8ae (DEFAULT_TABLE table=default.Table_fc10e7e4961046c1b63ae98f857f8ff1 [id=2a450b3771684460b7675234267cd2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.821179 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 622cc5f673694546a594f968621f1011 (DEFAULT_TABLE table=default.Table_cbf320b748bb4162a3ce1ef93588df7b [id=5ee3e2262ce0468bb5a53579df41168b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.821461 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413341019ff94a10896d956f5a03c8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.821609 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 622cc5f673694546a594f968621f1011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.821630 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 622cc5f673694546a594f968621f1011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.822129 1033 tablet_service.cc:1467] Processing CreateTablet for tablet a8fe6a54d4184b27928ddb02f188a9f9 (DEFAULT_TABLE table=default.Table_615f5ae1a3c84fbd85db081530862f20 [id=d3cbbb12f8ed4614a52af20b815c33d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.822585 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8fe6a54d4184b27928ddb02f188a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.824937 883 tablet_service.cc:1467] Processing CreateTablet for tablet 413341019ff94a10896d956f5a03c8ae (DEFAULT_TABLE table=default.Table_fc10e7e4961046c1b63ae98f857f8ff1 [id=2a450b3771684460b7675234267cd2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.825294 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413341019ff94a10896d956f5a03c8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.827313 747 tablet_service.cc:1467] Processing CreateTablet for tablet 622cc5f673694546a594f968621f1011 (DEFAULT_TABLE table=default.Table_cbf320b748bb4162a3ce1ef93588df7b [id=5ee3e2262ce0468bb5a53579df41168b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.827277 756 tablet_service.cc:1467] Processing CreateTablet for tablet 413341019ff94a10896d956f5a03c8ae (DEFAULT_TABLE table=default.Table_fc10e7e4961046c1b63ae98f857f8ff1 [id=2a450b3771684460b7675234267cd2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:03.827611 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 622cc5f673694546a594f968621f1011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.827777 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413341019ff94a10896d956f5a03c8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:03.834517 1160 log.cc:1198] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/cd596614521a4a92a71b781b162f70b6
I20250109 20:53:03.834889 1160 ts_tablet_manager.cc:1939] T cd596614521a4a92a71b781b162f70b6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:03.835565 1160 tablet_replica.cc:331] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:03.836071 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet cd596614521a4a92a71b781b162f70b6 (table default.Table_0e564cfa1eb74fd8849dc79656731710 [id=3b0671ce903343a0836151d4e140ddb1]) successfully deleted
I20250109 20:53:03.836257 1160 raft_consensus.cc:2238] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:03.836337 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9f38f01144874befa5c38a7f6af2d1be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f38f01144874befa5c38a7f6af2d1be already in progress: creating tablet
I20250109 20:53:03.836534 1160 raft_consensus.cc:2267] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.837123 1160 ts_tablet_manager.cc:1905] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.839650 1165 tablet_bootstrap.cc:492] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.839812 1165 ts_tablet_manager.cc:1397] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.386s user 0.010s sys 0.005s
I20250109 20:53:03.840682 1165 raft_consensus.cc:357] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.840948 1165 raft_consensus.cc:383] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.841037 1165 raft_consensus.cc:738] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.841292 1165 consensus_queue.cc:260] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.809160 1166 tablet_replica.cc:331] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:03.851366 1166 raft_consensus.cc:2238] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.851511 1165 ts_tablet_manager.cc:1428] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250109 20:53:03.851706 1166 raft_consensus.cc:2267] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.851826 1165 tablet_bootstrap.cc:492] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:03.852586 1166 ts_tablet_manager.cc:1905] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.864267 2032 raft_consensus.cc:491] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.864388 2032 raft_consensus.cc:513] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:03.865108 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 670375bfd5114040aed6be25d1374d42 consensus state has no leader
W20250109 20:53:03.865607 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7357b4f67dc6448dbb8f77dce81731f3 consensus state has no leader
I20250109 20:53:03.866451 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c9f7274a7542e89a19bd4afcf89205 with delete_type TABLET_DATA_DELETED (replaced by tablet 56fac585b2904ab490a6882dd5252a9a) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.870769 887 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e4cfb5a07841828e5206d100b4f835 with delete_type TABLET_DATA_DELETED (replaced by tablet 3d5e72e3aaea4360b7c8f619a831c1f1) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.878620 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da78917f0b4931ab1705371c1fe5dd with delete_type TABLET_DATA_DELETED (replaced by tablet 3974c4196d4043be9ad6e0452ed9cb7b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.883297 1176 tablet_bootstrap.cc:492] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.883374 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ddeb6e2ad0844a41b5aeaf86765e188e" } modify_external_catalogs: true
I20250109 20:53:03.883461 1176 ts_tablet_manager.cc:1397] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.113s user 0.002s sys 0.022s
I20250109 20:53:03.883523 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ddeb6e2ad0844a41b5aeaf86765e188e" } modify_external_catalogs: true
I20250109 20:53:03.883705 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_534986b560ef4cbdaf8e21e045a76d29" } new_table_name: "default.Table_e74b979d9894464eb19f40431eab462a" modify_external_catalogs: true
I20250109 20:53:03.883858 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_89576f5c757a4cc788608386bc1f949b" } modify_external_catalogs: true
I20250109 20:53:03.883975 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_89576f5c757a4cc788608386bc1f949b" } modify_external_catalogs: true
I20250109 20:53:03.884111 1176 raft_consensus.cc:357] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.884261 1176 raft_consensus.cc:383] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.884302 1176 raft_consensus.cc:738] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.884497 1176 consensus_queue.cc:260] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.884954 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4b2efcf7340c405483d625749dea769b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.885404 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b2efcf7340c405483d625749dea769b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.886967 2413 raft_consensus.cc:491] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.887107 2413 raft_consensus.cc:513] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.887290 1176 ts_tablet_manager.cc:1428] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:03.887537 1176 tablet_bootstrap.cc:492] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.887537 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5157484d0a55469184fc46834bcf66bb" } modify_external_catalogs: true
I20250109 20:53:03.887696 2413 leader_election.cc:290] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.887676 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5157484d0a55469184fc46834bcf66bb" } modify_external_catalogs: true
I20250109 20:53:03.888291 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ed5f7d5524c54a8debe1bc610bd25" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.888695 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_eff4d27a17b3425c92c3d388ee1cc73e" } modify_external_catalogs: true
I20250109 20:53:03.888868 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_eff4d27a17b3425c92c3d388ee1cc73e" } modify_external_catalogs: true
W20250109 20:53:03.888885 712 leader_election.cc:343] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.889343 1176 tablet_bootstrap.cc:654] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:03.889887 713 leader_election.cc:343] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 686ed5f7d5524c54a8debe1bc610bd25
I20250109 20:53:03.889989 713 leader_election.cc:304] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.890820 2413 raft_consensus.cc:2743] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.892544 2032 leader_election.cc:290] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.893131 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e9b31b27949c2948cb64b6124ee72" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:03.894047 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e9b31b27949c2948cb64b6124ee72" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:03.895601 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 404ee892ab79427baaef6d19c57699d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.904506 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 72cc0bbbb5584f14ba4d6c62211a12c5 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:03.905395 1158 tablet_bootstrap.cc:492] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:03.905560 1158 ts_tablet_manager.cc:1397] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.094s user 0.006s sys 0.012s
I20250109 20:53:03.905812 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f0d09d4894634825ad3aa7933bfb6010
I20250109 20:53:03.906299 1158 raft_consensus.cc:357] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.906472 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 21dbbfa0af5b4f0487dd5d9db8efaeda
I20250109 20:53:03.906520 1158 raft_consensus.cc:383] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.906569 1158 raft_consensus.cc:738] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.906771 1158 consensus_queue.cc:260] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.894532 854 leader_election.cc:304] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.907447 2032 raft_consensus.cc:2743] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.908607 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:03.910032 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ece9f0afb8c24a1f95409d21d052a57b consensus state has no leader
I20250109 20:53:03.911077 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7b8344dbb1c540abadd22531e19a52d1
I20250109 20:53:03.912000 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b8344dbb1c540abadd22531e19a52d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.912422 1158 ts_tablet_manager.cc:1428] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250109 20:53:03.912667 1158 tablet_bootstrap.cc:492] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:03.912937 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6c4abc19297e4029b1b7c5396b49b967
I20250109 20:53:03.913867 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4abc19297e4029b1b7c5396b49b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.913923 1158 tablet_bootstrap.cc:654] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.914244 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b8344dbb1c540abadd22531e19a52d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.914582 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4abc19297e4029b1b7c5396b49b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.914878 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4abc19297e4029b1b7c5396b49b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.915608 1165 tablet_bootstrap.cc:654] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.919059 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:03.920091 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f420b7e623243e093d26c95276c5af9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2420b285905342b3820f9549c8f0c4ad) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.920441 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_47a796218c0848d9990864cc20f7ae05" } modify_external_catalogs: true
I20250109 20:53:03.920575 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_47a796218c0848d9990864cc20f7ae05" } modify_external_catalogs: true
I20250109 20:53:03.924039 2032 raft_consensus.cc:491] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.924168 2032 raft_consensus.cc:513] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.924739 2032 leader_election.cc:290] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:03.926163 854 leader_election.cc:343] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 35a39fac67894a208dd7ec3153b05e4a
W20250109 20:53:03.926353 854 leader_election.cc:343] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 35a39fac67894a208dd7ec3153b05e4a
I20250109 20:53:03.926450 854 leader_election.cc:304] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.926709 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d09d4894634825ad3aa7933bfb6010 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.926811 2032 raft_consensus.cc:2743] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.926901 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d09d4894634825ad3aa7933bfb6010 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.927129 745 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d09d4894634825ad3aa7933bfb6010 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.913841 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b8344dbb1c540abadd22531e19a52d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.927429 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.927624 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.928591 1157 ts_tablet_manager.cc:1918] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.928889 1157 log.cc:1198] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/72cc0bbbb5584f14ba4d6c62211a12c5
I20250109 20:53:03.929239 1157 ts_tablet_manager.cc:1939] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:03.930562 1166 ts_tablet_manager.cc:1918] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.931309 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1187) took 1024 ms. Trace:
I20250109 20:53:03.931438 1157 rpcz_store.cc:276] 0109 20:53:02.906501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.906550 (+ 49us) service_pool.cc:225] Handling call
0109 20:53:03.755077 (+848527us) ts_tablet_manager.cc:1145] Deleting tablet 72cc0bbbb5584f14ba4d6c62211a12c5
0109 20:53:03.755085 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.860399 (+105314us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.928500 (+ 68101us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.931294 (+ 2794us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":848378}
I20250109 20:53:03.932161 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 72cc0bbbb5584f14ba4d6c62211a12c5 (table default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]) successfully deleted
W20250109 20:53:03.932747 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9f38f01144874befa5c38a7f6af2d1be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f38f01144874befa5c38a7f6af2d1be already in progress: creating tablet
I20250109 20:53:03.932984 1166 log.cc:1198] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/bd62a3ae64a143998bb1172b3a97c69f
I20250109 20:53:03.933260 1166 ts_tablet_manager.cc:1939] T bd62a3ae64a143998bb1172b3a97c69f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:03.933357 1158 tablet_bootstrap.cc:492] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
W20250109 20:53:03.933616 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 17940f311b0440a59a42cc57f3a0bc4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17940f311b0440a59a42cc57f3a0bc4c already in progress: creating tablet
I20250109 20:53:03.934150 1176 tablet_bootstrap.cc:492] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.934309 1176 ts_tablet_manager.cc:1397] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.006s
I20250109 20:53:03.934554 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1165) took 1057 ms. Trace:
I20250109 20:53:03.934746 1166 rpcz_store.cc:276] 0109 20:53:02.877051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.882336 (+ 5285us) service_pool.cc:225] Handling call
0109 20:53:03.809107 (+926771us) ts_tablet_manager.cc:1145] Deleting tablet bd62a3ae64a143998bb1172b3a97c69f
0109 20:53:03.809118 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.880237 (+ 71119us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.930477 (+ 50240us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.934538 (+ 4061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926557}
I20250109 20:53:03.935307 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet bd62a3ae64a143998bb1172b3a97c69f (table default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]) successfully deleted
I20250109 20:53:03.935595 1166 tablet_replica.cc:331] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:03.935611 1176 raft_consensus.cc:357] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.935911 1166 raft_consensus.cc:2238] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.936026 1158 ts_tablet_manager.cc:1397] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20250109 20:53:03.936100 1166 raft_consensus.cc:2267] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.933914 1157 tablet_replica.cc:331] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:03.936934 1157 raft_consensus.cc:2238] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.937096 1157 raft_consensus.cc:2267] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.937575 1157 ts_tablet_manager.cc:1905] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.935916 1176 raft_consensus.cc:383] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.938970 1160 ts_tablet_manager.cc:1918] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.939029 1176 raft_consensus.cc:738] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.939062 1160 log.cc:1198] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/e0f67c4294f8499496db4e82db8c3bf0
I20250109 20:53:03.939368 1160 ts_tablet_manager.cc:1939] T e0f67c4294f8499496db4e82db8c3bf0 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:03.939268 1176 consensus_queue.cc:260] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.939708 1176 ts_tablet_manager.cc:1428] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:53:03.940095 1166 ts_tablet_manager.cc:1905] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.940312 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet e0f67c4294f8499496db4e82db8c3bf0 (table default.Table_bf29c4a64f7b4d49af314ad6211982b1 [id=7d02be4e89a7426ea835a9041c394fd4]) successfully deleted
I20250109 20:53:03.940594 2413 raft_consensus.cc:491] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.940693 2413 raft_consensus.cc:513] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.940989 1176 tablet_bootstrap.cc:492] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:03.941258 2413 leader_election.cc:290] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.941346 2413 raft_consensus.cc:491] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.941421 2413 raft_consensus.cc:513] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.941846 2413 leader_election.cc:290] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.942395 1176 tablet_bootstrap.cc:654] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.942364 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80da78917f0b4931ab1705371c1fe5dd" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.942710 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e090f72eb1e046319b825ee5fe126332" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:03.942842 712 leader_election.cc:343] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:03.943132 712 leader_election.cc:343] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:03.944020 713 leader_election.cc:343] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 80da78917f0b4931ab1705371c1fe5dd
I20250109 20:53:03.944115 713 leader_election.cc:304] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.944311 2417 raft_consensus.cc:2743] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:03.944469 713 leader_election.cc:343] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: e090f72eb1e046319b825ee5fe126332
I20250109 20:53:03.944559 713 leader_election.cc:304] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.944751 2417 raft_consensus.cc:2743] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:03.945256 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 70774.4us
I20250109 20:53:03.951426 1166 ts_tablet_manager.cc:1918] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.951542 1166 log.cc:1198] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/006475691733470f8a0468365972dfd2
I20250109 20:53:03.951646 2417 raft_consensus.cc:491] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:03.951844 1166 ts_tablet_manager.cc:1939] T 006475691733470f8a0468365972dfd2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:03.951776 2417 raft_consensus.cc:513] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.952404 2417 leader_election.cc:290] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:03.952600 1158 raft_consensus.cc:357] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.952991 1158 raft_consensus.cc:383] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.953214 1158 raft_consensus.cc:738] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.953405 1158 consensus_queue.cc:260] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.953485 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1164) took 1075 ms. Trace:
I20250109 20:53:03.953486 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246fd9ac96f54d008e8203e4580e7ef3" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:03.953559 1166 rpcz_store.cc:276] 0109 20:53:02.876946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.885169 (+ 8223us) service_pool.cc:225] Handling call
0109 20:53:03.935554 (+1050385us) ts_tablet_manager.cc:1145] Deleting tablet 006475691733470f8a0468365972dfd2
0109 20:53:03.935564 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.947109 (+ 11545us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.951324 (+ 4215us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.952463 (+ 1139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049966}
I20250109 20:53:03.953899 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 006475691733470f8a0468365972dfd2 (table default.Table_d5db66989ced4d6fb84352f19e087cc7 [id=d65cb2a049bb450391b2f32bb6bbeead]) successfully deleted
I20250109 20:53:03.954135 1166 tablet_replica.cc:331] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:03.954376 1166 raft_consensus.cc:2238] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.954526 1166 raft_consensus.cc:2267] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.955054 1166 ts_tablet_manager.cc:1905] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:03.953779 713 leader_election.cc:343] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 246fd9ac96f54d008e8203e4580e7ef3
W20250109 20:53:03.956945 712 leader_election.cc:343] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:03.957126 712 leader_election.cc:304] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:03.957480 2417 raft_consensus.cc:2743] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:03.958014 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e1faf5d79a9b43ffafc9c34a5937fcc0
I20250109 20:53:03.960965 1166 ts_tablet_manager.cc:1918] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.961053 1166 log.cc:1198] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/0ec1d30b67674facb8b92a2c1a408073
I20250109 20:53:03.961316 1166 ts_tablet_manager.cc:1939] T 0ec1d30b67674facb8b92a2c1a408073 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:03.963987 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1167) took 1071 ms. Trace:
I20250109 20:53:03.964087 1166 rpcz_store.cc:276] 0109 20:53:02.892200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.892766 (+ 566us) service_pool.cc:225] Handling call
0109 20:53:03.954096 (+1061330us) ts_tablet_manager.cc:1145] Deleting tablet 0ec1d30b67674facb8b92a2c1a408073
0109 20:53:03.954103 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.958151 (+ 4048us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.960900 (+ 2749us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:03.963974 (+ 3074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061186}
I20250109 20:53:03.962200 1157 ts_tablet_manager.cc:1918] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:03.964483 1157 log.cc:1198] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/686ed5f7d5524c54a8debe1bc610bd25
I20250109 20:53:03.964781 1157 ts_tablet_manager.cc:1939] T 686ed5f7d5524c54a8debe1bc610bd25 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:03.966156 1157 tablet_replica.cc:331] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:03.963923 750 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.966468 1157 raft_consensus.cc:2238] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:03.966699 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1166) took 1074 ms. Trace:
I20250109 20:53:03.966820 1157 raft_consensus.cc:2267] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.962936 1158 ts_tablet_manager.cc:1428] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250109 20:53:03.964430 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 0ec1d30b67674facb8b92a2c1a408073 (table default.Table_0cbc3b8995704eaf99466fd1d5d0e124 [id=22ffbc424e6d4038a66ddea1fca9842e]) successfully deleted
I20250109 20:53:03.968816 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 686ed5f7d5524c54a8debe1bc610bd25 (table default.Table_6cca94c4ba974861b95038c43ebf833a [id=bf679a467a78433899ead891f63fed13]) successfully deleted
W20250109 20:53:03.968999 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e0f47f9b1f8a479a9760526a4873af57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f47f9b1f8a479a9760526a4873af57 already in progress: creating tablet
W20250109 20:53:03.969193 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e80959e8c0944167b4ddebf4ebb382c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e80959e8c0944167b4ddebf4ebb382c5 already in progress: creating tablet
I20250109 20:53:03.969748 1158 tablet_bootstrap.cc:492] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:03.971252 1158 tablet_bootstrap.cc:654] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:03.972016 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f2ba81d736745fb9d56fd9a7c229153 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:03.972939 1166 rpcz_store.cc:276] 0109 20:53:02.892107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:02.892763 (+ 656us) service_pool.cc:225] Handling call
0109 20:53:03.966587 (+1073824us) ts_tablet_manager.cc:1145] Deleting tablet 90bcfd699e094c499df625281e7119b1
0109 20:53:03.966596 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:03.966683 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069700}
W20250109 20:53:03.973321 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 90bcfd699e094c499df625281e7119b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90bcfd699e094c499df625281e7119b1 already in progress: creating tablet
I20250109 20:53:03.974462 750 tablet_service.cc:1514] Processing DeleteTablet for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.975047 887 tablet_service.cc:1514] Processing DeleteTablet for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:03.975531 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9f38f01144874befa5c38a7f6af2d1be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f38f01144874befa5c38a7f6af2d1be already in progress: creating tablet
I20250109 20:53:03.977772 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.979298 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8f2ba81d736745fb9d56fd9a7c229153
I20250109 20:53:03.979745 451 catalog_manager.cc:6596] ReplaceTablet: tablet 8f2ba81d736745fb9d56fd9a7c229153 deleted and replaced by tablet 49e0863249364194936804ae83645936
I20250109 20:53:03.980947 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2ba81d736745fb9d56fd9a7c229153 with delete_type TABLET_DATA_DELETED (replaced by tablet 49e0863249364194936804ae83645936) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:03.981177 1166 tablet_replica.cc:331] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:03.981346 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2ba81d736745fb9d56fd9a7c229153 with delete_type TABLET_DATA_DELETED (replaced by tablet 49e0863249364194936804ae83645936) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:03.981729 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2ba81d736745fb9d56fd9a7c229153 with delete_type TABLET_DATA_DELETED (replaced by tablet 49e0863249364194936804ae83645936) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:03.982923 1157 ts_tablet_manager.cc:1905] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.985723 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_18a1b7ae7954468a851982d70404d1ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.986254 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18a1b7ae7954468a851982d70404d1ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.985769 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b66c546088d141c59f36c5a8a911fb1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.987392 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b66c546088d141c59f36c5a8a911fb1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.987474 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_91c5f2662902438187bc75ecc62c5017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.987823 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91c5f2662902438187bc75ecc62c5017 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.981351 1166 raft_consensus.cc:2238] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:03.988873 1166 raft_consensus.cc:2267] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.989418 1166 ts_tablet_manager.cc:1905] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.990034 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3074f0a98179449ab995c6544812e329" } new_table_name: "default.Table_8ec506c30a204667919a43f12ab10a09" modify_external_catalogs: true
I20250109 20:53:03.990510 1157 ts_tablet_manager.cc:1918] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:03.990613 1157 log.cc:1198] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/be44ffc9e371452f8d5f4174fbfa752a
I20250109 20:53:03.990939 1157 ts_tablet_manager.cc:1939] T be44ffc9e371452f8d5f4174fbfa752a P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:03.991088 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b49ca6bc758a4b8c9c39cc064d3e44c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.991432 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b49ca6bc758a4b8c9c39cc064d3e44c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.992321 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_22d4e51e64bb4059ab767ed4d99798b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:03.992679 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22d4e51e64bb4059ab767ed4d99798b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:03.993572 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_50d70e8a583c47059eafc658d73feb69 [id=ee4103ad12894523a9f260bc6ff91419]) successfully deleted
I20250109 20:53:03.994632 1157 tablet_replica.cc:331] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:03.994800 1157 raft_consensus.cc:2238] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:03.994951 1165 tablet_bootstrap.cc:492] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:03.995052 1157 raft_consensus.cc:2267] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:03.995080 1165 ts_tablet_manager.cc:1397] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.143s user 0.002s sys 0.004s
I20250109 20:53:03.995575 1157 ts_tablet_manager.cc:1905] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:03.995589 1160 tablet_replica.cc:331] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:03.995820 1165 raft_consensus.cc:357] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.996389 1165 raft_consensus.cc:383] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:03.996516 1165 raft_consensus.cc:738] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:03.996815 1165 consensus_queue.cc:260] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:03.999198 1176 tablet_bootstrap.cc:492] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:03.999405 1176 ts_tablet_manager.cc:1397] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.006s
I20250109 20:53:04.000598 1176 raft_consensus.cc:357] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.000946 1176 raft_consensus.cc:383] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.001087 1176 raft_consensus.cc:738] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.001423 1176 consensus_queue.cc:260] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.003471 1157 ts_tablet_manager.cc:1918] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:04.003567 1157 log.cc:1198] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/06e615ce8e124cfe9432778b6b58e117
I20250109 20:53:04.003867 1157 ts_tablet_manager.cc:1939] T 06e615ce8e124cfe9432778b6b58e117 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.004835 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 06e615ce8e124cfe9432778b6b58e117 (table default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]) successfully deleted
I20250109 20:53:04.010319 1157 tablet_replica.cc:331] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:53:04.010430 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4cff74250cd435e8a52c010c11eff9b has no consensus state
I20250109 20:53:04.010680 1157 raft_consensus.cc:2238] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.011077 1157 raft_consensus.cc:2267] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.011158 1166 ts_tablet_manager.cc:1918] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.011796 1157 ts_tablet_manager.cc:1905] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.013057 2032 raft_consensus.cc:491] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.013193 2032 raft_consensus.cc:513] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.013713 446 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:04.013752 2032 leader_election.cc:290] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:04.015275 854 leader_election.cc:343] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 2ca83a49523b45be9ac7556b40746a83
W20250109 20:53:04.015458 854 leader_election.cc:343] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2ca83a49523b45be9ac7556b40746a83
I20250109 20:53:04.015535 854 leader_election.cc:304] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.015947 2032 raft_consensus.cc:2743] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.018476 1158 tablet_bootstrap.cc:492] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:04.018638 1158 ts_tablet_manager.cc:1397] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.010s
I20250109 20:53:04.019296 1158 raft_consensus.cc:357] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.019363 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_848a8bb910414b7faca9f9f52cbfc502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.019519 1158 raft_consensus.cc:383] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.019573 1158 raft_consensus.cc:738] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:04.019817 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_848a8bb910414b7faca9f9f52cbfc502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.019785 1158 consensus_queue.cc:260] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.021032 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e79e938de6ba4b929f53a7f9656e8416" } new_table_name: "default.Table_800bb021710b4f01a0769f08ca4f5fb1" modify_external_catalogs: true
I20250109 20:53:04.021759 1166 log.cc:1198] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6569b450bb7a4a78922061d2ede981fe
I20250109 20:53:04.019719 1176 ts_tablet_manager.cc:1428] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250109 20:53:04.022195 1176 tablet_bootstrap.cc:492] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:04.022861 1166 ts_tablet_manager.cc:1939] T 6569b450bb7a4a78922061d2ede981fe P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:04.023448 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_502dc59bfc324b38ba72180fd2141f62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.023914 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_502dc59bfc324b38ba72180fd2141f62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.027849 1166 tablet_replica.cc:331] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:04.027866 750 tablet_service.cc:1514] Processing DeleteTablet for tablet bba303032679462c8b16f706096b6d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:04.028028 1166 raft_consensus.cc:2238] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.029563 1176 tablet_bootstrap.cc:654] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.029964 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_93ab49fd9374420f80e0422073c909da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.030179 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6569b450bb7a4a78922061d2ede981fe (table default.Table_416c9e793e9d4fa69bd29e1e30365e06 [id=ee1cb175e24b4968b58f5d87eaeeefb5]) successfully deleted
I20250109 20:53:04.030441 1166 raft_consensus.cc:2267] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:04.030536 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 17940f311b0440a59a42cc57f3a0bc4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17940f311b0440a59a42cc57f3a0bc4c already in progress: creating tablet
I20250109 20:53:04.030786 1158 ts_tablet_manager.cc:1428] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:53:04.030903 1166 ts_tablet_manager.cc:1905] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:04.031004 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93ab49fd9374420f80e0422073c909da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:04.034091 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 049321d28ab241138247dcefa0411f54 consensus state has no leader
I20250109 20:53:04.034618 2417 raft_consensus.cc:491] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.034735 2417 raft_consensus.cc:513] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.035306 2417 leader_election.cc:290] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:04.036129 713 leader_election.cc:343] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: b68d23fb2df743f4afa2682f6d2e3efa
I20250109 20:53:04.036140 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b68d23fb2df743f4afa2682f6d2e3efa" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:04.036635 712 leader_election.cc:343] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.036725 712 leader_election.cc:304] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.037393 2417 raft_consensus.cc:2743] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.037654 1165 ts_tablet_manager.cc:1428] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20250109 20:53:04.041071 451 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:04.043846 1158 tablet_bootstrap.cc:492] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:04.046687 1165 tablet_bootstrap.cc:492] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:04.048040 1165 tablet_bootstrap.cc:654] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.048274 1166 ts_tablet_manager.cc:1918] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.048363 1166 log.cc:1198] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/9d7587432d7345e4a225fc7d5c968c27
I20250109 20:53:04.059538 1158 tablet_bootstrap.cc:654] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.060917 1160 raft_consensus.cc:2238] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.061138 1160 raft_consensus.cc:2267] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.061731 1160 ts_tablet_manager.cc:1905] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.062139 1166 ts_tablet_manager.cc:1939] T 9d7587432d7345e4a225fc7d5c968c27 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:04.063189 1157 ts_tablet_manager.cc:1918] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.064775 1157 log.cc:1198] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/246fd9ac96f54d008e8203e4580e7ef3
I20250109 20:53:04.065119 1157 ts_tablet_manager.cc:1939] T 246fd9ac96f54d008e8203e4580e7ef3 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.065796 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 9d7587432d7345e4a225fc7d5c968c27 (table default.Table_b590793190454bb09e3e7dd36515f65a [id=4a57bf26eda04187bd4b854d402f3c19]) successfully deleted
I20250109 20:53:04.067327 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dfc57ba40e274837bd12f0b1085b2eed" } new_table_name: "default.Table_b5e396ef02d74355a6735e5095163c92" modify_external_catalogs: true
I20250109 20:53:04.067487 2368 raft_consensus.cc:491] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.067600 2368 raft_consensus.cc:513] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.069267 2368 leader_election.cc:290] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.070173 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4814361a1e854451a0ee7469cc6d447b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.070542 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4814361a1e854451a0ee7469cc6d447b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:04.071131 989 leader_election.cc:343] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:04.071486 990 leader_election.cc:343] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.071570 990 leader_election.cc:304] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.071945 2368 raft_consensus.cc:2743] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.072451 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 246fd9ac96f54d008e8203e4580e7ef3 (table default.Table_8ed167dde7b94790b7731c10aa30628c [id=87a978ea0af846f0b3ea56a3d1942d6d]) successfully deleted
I20250109 20:53:04.072904 1157 tablet_replica.cc:331] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:04.073161 1157 raft_consensus.cc:2238] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.073410 1157 raft_consensus.cc:2267] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.073451 1176 tablet_bootstrap.cc:492] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:04.073628 1176 ts_tablet_manager.cc:1397] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.001s
I20250109 20:53:04.074040 1157 ts_tablet_manager.cc:1905] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:04.074451 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51b760fcb4fc466ba49d456892d43f3c consensus state has no leader
I20250109 20:53:04.074491 1176 raft_consensus.cc:357] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.074695 1176 raft_consensus.cc:383] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.074787 1176 raft_consensus.cc:738] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.075039 1176 consensus_queue.cc:260] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.075296 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c0d1dadec3324eee8cc42a24ab27f3ed" } new_table_name: "default.Table_0964845bb53b4ea5b1962664a945a369" modify_external_catalogs: true
I20250109 20:53:04.075846 1176 ts_tablet_manager.cc:1428] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250109 20:53:04.079559 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b17962015e04c6b85169915586334cc consensus state has no leader
W20250109 20:53:04.085160 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e0f47f9b1f8a479a9760526a4873af57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f47f9b1f8a479a9760526a4873af57 already in progress: creating tablet
I20250109 20:53:04.087564 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_83001a14d61f44ec8e047a6fe2dccd69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.088050 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e80959e8c0944167b4ddebf4ebb382c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e80959e8c0944167b4ddebf4ebb382c5 already in progress: creating tablet
W20250109 20:53:04.088164 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83001a14d61f44ec8e047a6fe2dccd69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.088718 1166 tablet_replica.cc:331] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:04.088847 2413 raft_consensus.cc:491] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.088961 2413 raft_consensus.cc:513] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.089566 1176 tablet_bootstrap.cc:492] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:04.089567 2413 leader_election.cc:290] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:04.090538 713 leader_election.cc:343] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: bba303032679462c8b16f706096b6d4d
I20250109 20:53:04.090803 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba303032679462c8b16f706096b6d4d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:04.091302 712 leader_election.cc:343] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.091392 712 leader_election.cc:304] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.091743 2413 raft_consensus.cc:2743] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.091799 1176 tablet_bootstrap.cc:654] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.092973 1166 raft_consensus.cc:2238] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.094305 1166 raft_consensus.cc:2267] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.096601 1158 tablet_bootstrap.cc:492] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:04.096812 1158 ts_tablet_manager.cc:1397] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250109 20:53:04.097493 1158 raft_consensus.cc:357] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.097718 1158 raft_consensus.cc:383] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.097815 1158 raft_consensus.cc:738] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.097855 1166 ts_tablet_manager.cc:1905] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.098073 1158 consensus_queue.cc:260] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.106673 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 225046d45c1c43b488babf5da8de75d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:04.107578 1158 ts_tablet_manager.cc:1428] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250109 20:53:04.108098 1165 tablet_bootstrap.cc:492] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:04.108306 1165 ts_tablet_manager.cc:1397] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.001s
I20250109 20:53:04.108356 1158 tablet_bootstrap.cc:492] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:04.109264 1165 raft_consensus.cc:357] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.109519 1165 raft_consensus.cc:383] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.109611 1165 raft_consensus.cc:738] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.109841 1176 tablet_bootstrap.cc:492] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:04.109970 1158 tablet_bootstrap.cc:654] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.109885 1165 consensus_queue.cc:260] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.110011 1176 ts_tablet_manager.cc:1397] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20250109 20:53:04.110597 1176 raft_consensus.cc:357] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.110807 1176 raft_consensus.cc:383] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.110899 1176 raft_consensus.cc:738] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.111128 1176 consensus_queue.cc:260] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.113459 1176 ts_tablet_manager.cc:1428] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:53:04.115150 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e82535780c47ca8b9cd94b013b4b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.119647 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7fb50e2939fb4fe39c63669d58697188" } modify_external_catalogs: true
I20250109 20:53:04.119823 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7fb50e2939fb4fe39c63669d58697188" } modify_external_catalogs: true
I20250109 20:53:04.120380 1176 tablet_bootstrap.cc:492] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:04.120561 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bd1103b105f94221b1bed1edfe6477e3" } new_table_name: "default.Table_10f45fa0e77c4c79a77da8ec132dc497" modify_external_catalogs: true
I20250109 20:53:04.122025 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_aff5b59bbc5445d8b655981c6f4ca537" } modify_external_catalogs: true
I20250109 20:53:04.122170 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_aff5b59bbc5445d8b655981c6f4ca537" } modify_external_catalogs: true
I20250109 20:53:04.123095 1176 tablet_bootstrap.cc:654] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.125483 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ff10bec6ea2f476894eda188f6b9c63d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.126057 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff10bec6ea2f476894eda188f6b9c63d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.127542 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d09bd36c6dd447228bb30f6fa48217e8" } modify_external_catalogs: true
I20250109 20:53:04.127712 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d09bd36c6dd447228bb30f6fa48217e8" } modify_external_catalogs: true
I20250109 20:53:04.127907 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6de1d3cd236842a3a239ec4c275d4986" } modify_external_catalogs: true
I20250109 20:53:04.128029 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6de1d3cd236842a3a239ec4c275d4986" } modify_external_catalogs: true
I20250109 20:53:04.128095 893 tablet_service.cc:1467] Processing CreateTablet for tablet 563c1fcad384470a9f7663d48ad52312 (DEFAULT_TABLE table=default.Table_2e1e70e4ec454bb19ce9f47d5272e0e0 [id=55be443448804444b5b82a22cb8ea586]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.128484 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563c1fcad384470a9f7663d48ad52312. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.134171 1157 ts_tablet_manager.cc:1918] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.134984 1157 log.cc:1198] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4160bb4ba40347bda48f8fa88f8b6001
I20250109 20:53:04.135896 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 563c1fcad384470a9f7663d48ad52312 (DEFAULT_TABLE table=default.Table_2e1e70e4ec454bb19ce9f47d5272e0e0 [id=55be443448804444b5b82a22cb8ea586]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.136319 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563c1fcad384470a9f7663d48ad52312. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.136425 745 tablet_service.cc:1467] Processing CreateTablet for tablet 2871457499b64529851a4ec12e80838d (DEFAULT_TABLE table=default.Table_9657151670d0422b879b12781fed0eb5 [id=9b19f3bb83954c348e575ac4cdd0e38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.136823 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2871457499b64529851a4ec12e80838d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.137985 750 tablet_service.cc:1467] Processing CreateTablet for tablet 563c1fcad384470a9f7663d48ad52312 (DEFAULT_TABLE table=default.Table_2e1e70e4ec454bb19ce9f47d5272e0e0 [id=55be443448804444b5b82a22cb8ea586]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.138239 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563c1fcad384470a9f7663d48ad52312. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.141326 887 tablet_service.cc:1467] Processing CreateTablet for tablet 2871457499b64529851a4ec12e80838d (DEFAULT_TABLE table=default.Table_9657151670d0422b879b12781fed0eb5 [id=9b19f3bb83954c348e575ac4cdd0e38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.141660 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2871457499b64529851a4ec12e80838d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.142052 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04b4247f52ce4d33bf49d9625a1592bf consensus state has no leader
I20250109 20:53:04.134865 1158 tablet_bootstrap.cc:492] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:04.142246 1158 ts_tablet_manager.cc:1397] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250109 20:53:04.142903 1158 raft_consensus.cc:357] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.143088 1158 raft_consensus.cc:383] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.143138 1158 raft_consensus.cc:738] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.143343 1158 consensus_queue.cc:260] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.145793 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet c865faed94c3493ea638826bdcd4253d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.146013 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 4870a667cd3e410590923c9e2497af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.145884 1158 ts_tablet_manager.cc:1428] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:53:04.148808 1157 ts_tablet_manager.cc:1939] T 4160bb4ba40347bda48f8fa88f8b6001 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.150009 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7357b4f67dc6448dbb8f77dce81731f3
I20250109 20:53:04.150820 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 7357b4f67dc6448dbb8f77dce81731f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.150864 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7357b4f67dc6448dbb8f77dce81731f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.151362 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet cb880d4da1ca4c2c851fc8c1e0846ce4
I20250109 20:53:04.152269 890 tablet_service.cc:1514] Processing DeleteTablet for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.152326 1157 tablet_replica.cc:331] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:04.152637 882 tablet_service.cc:1467] Processing CreateTablet for tablet ebaaefdc13e84f2e98f9926278adf637 (DEFAULT_TABLE table=default.Table_c55a378736c64b6ea978572ddc8c6e4e [id=45ba0ad8885f41b69c16d84c9399a720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.152649 1157 raft_consensus.cc:2238] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.152918 1157 raft_consensus.cc:2267] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.152987 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebaaefdc13e84f2e98f9926278adf637. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.153002 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.153175 747 tablet_service.cc:1514] Processing DeleteTablet for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:04.153425 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 2871457499b64529851a4ec12e80838d (DEFAULT_TABLE table=default.Table_9657151670d0422b879b12781fed0eb5 [id=9b19f3bb83954c348e575ac4cdd0e38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.153532 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4160bb4ba40347bda48f8fa88f8b6001 (table default.Table_13f472ba51244613ab6c1c0cc5e9f9e4 [id=3fc24473de2845628a019824b932a4e6]) successfully deleted
I20250109 20:53:04.153682 1157 ts_tablet_manager.cc:1905] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.153725 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2871457499b64529851a4ec12e80838d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.154093 1158 tablet_bootstrap.cc:492] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:04.155261 1158 tablet_bootstrap.cc:654] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.156981 746 tablet_service.cc:1467] Processing CreateTablet for tablet 568e4a83fddd45c8b61849a369074c4d (DEFAULT_TABLE table=default.Table_70b2bddd5393467f8ae2ff2ea7d1a79d [id=854b3a463667412eb44babe52e3b7d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.157114 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 568e4a83fddd45c8b61849a369074c4d (DEFAULT_TABLE table=default.Table_70b2bddd5393467f8ae2ff2ea7d1a79d [id=854b3a463667412eb44babe52e3b7d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.157297 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568e4a83fddd45c8b61849a369074c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.157397 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568e4a83fddd45c8b61849a369074c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.160971 883 tablet_service.cc:1467] Processing CreateTablet for tablet 568e4a83fddd45c8b61849a369074c4d (DEFAULT_TABLE table=default.Table_70b2bddd5393467f8ae2ff2ea7d1a79d [id=854b3a463667412eb44babe52e3b7d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.161284 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568e4a83fddd45c8b61849a369074c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.161655 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d79437d800f44c5a8222eea7dc2cd4e3
I20250109 20:53:04.162236 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d79437d800f44c5a8222eea7dc2cd4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:04.163149 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet d79437d800f44c5a8222eea7dc2cd4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.163331 1158 tablet_bootstrap.cc:492] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:04.163473 1158 ts_tablet_manager.cc:1397] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20250109 20:53:04.164212 1158 raft_consensus.cc:357] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.164450 1158 raft_consensus.cc:383] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.164544 1158 raft_consensus.cc:738] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.164799 1158 consensus_queue.cc:260] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.164952 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d79437d800f44c5a8222eea7dc2cd4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.172659 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8e32080157a2410996431df18465f86e (DEFAULT_TABLE table=default.Table_dce6c039ff03443199818ad6d730929d [id=f2b8d8bcfda84ba1997279f0906143ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.172986 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4b7e008a474978b91426f63dbd96ee) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.173329 745 tablet_service.cc:1467] Processing CreateTablet for tablet ebaaefdc13e84f2e98f9926278adf637 (DEFAULT_TABLE table=default.Table_c55a378736c64b6ea978572ddc8c6e4e [id=45ba0ad8885f41b69c16d84c9399a720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.173576 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ac867ce748f749d88885ef13654983ec
I20250109 20:53:04.174016 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e32080157a2410996431df18465f86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.174782 893 tablet_service.cc:1514] Processing DeleteTablet for tablet ac867ce748f749d88885ef13654983ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.175197 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebaaefdc13e84f2e98f9926278adf637. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.176767 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 7357b4f67dc6448dbb8f77dce81731f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:04.178459 1020 tablet_service.cc:1467] Processing CreateTablet for tablet ebaaefdc13e84f2e98f9926278adf637 (DEFAULT_TABLE table=default.Table_c55a378736c64b6ea978572ddc8c6e4e [id=45ba0ad8885f41b69c16d84c9399a720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.179127 1165 ts_tablet_manager.cc:1428] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.071s user 0.001s sys 0.000s
I20250109 20:53:04.178767 1166 ts_tablet_manager.cc:1918] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:04.173686 746 tablet_service.cc:1467] Processing CreateTablet for tablet 8e32080157a2410996431df18465f86e (DEFAULT_TABLE table=default.Table_dce6c039ff03443199818ad6d730929d [id=f2b8d8bcfda84ba1997279f0906143ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.174464 747 tablet_service.cc:1514] Processing DeleteTablet for tablet ac867ce748f749d88885ef13654983ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:04.180711 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e32080157a2410996431df18465f86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.180287 1166 log.cc:1198] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/be44ffc9e371452f8d5f4174fbfa752a
I20250109 20:53:04.180946 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet ac867ce748f749d88885ef13654983ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.182299 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 8e32080157a2410996431df18465f86e (DEFAULT_TABLE table=default.Table_dce6c039ff03443199818ad6d730929d [id=f2b8d8bcfda84ba1997279f0906143ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.182358 893 tablet_service.cc:1467] Processing CreateTablet for tablet d9707ae0844b4c4ea63c95d8ccdfdb21 (DEFAULT_TABLE table=default.Table_80c12feb8e5f49aca8a5d784a324e389 [id=dee1fc7b81aa40bb9dc1a14fd6e405a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.180963 1158 ts_tablet_manager.cc:1428] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250109 20:53:04.182715 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9707ae0844b4c4ea63c95d8ccdfdb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.178838 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebaaefdc13e84f2e98f9926278adf637. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.185812 1166 ts_tablet_manager.cc:1939] T be44ffc9e371452f8d5f4174fbfa752a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:04.187055 747 tablet_service.cc:1467] Processing CreateTablet for tablet d9707ae0844b4c4ea63c95d8ccdfdb21 (DEFAULT_TABLE table=default.Table_80c12feb8e5f49aca8a5d784a324e389 [id=dee1fc7b81aa40bb9dc1a14fd6e405a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.187357 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9707ae0844b4c4ea63c95d8ccdfdb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.191087 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d9707ae0844b4c4ea63c95d8ccdfdb21 (DEFAULT_TABLE table=default.Table_80c12feb8e5f49aca8a5d784a324e389 [id=dee1fc7b81aa40bb9dc1a14fd6e405a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.191336 882 tablet_service.cc:1467] Processing CreateTablet for tablet 71db3cb472bd4534a2efe1918cec848f (DEFAULT_TABLE table=default.Table_d74b3ee84e164e75a624c43034693fc9 [id=6b4cf82c617e4e7092ff5ee9145382ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.191435 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9707ae0844b4c4ea63c95d8ccdfdb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.191620 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71db3cb472bd4534a2efe1918cec848f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.192081 712 consensus_peers.cc:487] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:04.194314 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e32080157a2410996431df18465f86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.197428 1158 tablet_bootstrap.cc:492] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:04.197698 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_50d70e8a583c47059eafc658d73feb69 [id=ee4103ad12894523a9f260bc6ff91419]) successfully deleted
I20250109 20:53:04.197782 1165 tablet_bootstrap.cc:492] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:04.198689 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0075c64fe3164e5688ab5f2c13c736e6 (DEFAULT_TABLE table=default.Table_d4c1d0cf8d7f48bfbe31ec25d79cd5d2 [id=3551c51ff33b4789b8e5d733e5ac3d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.199064 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0075c64fe3164e5688ab5f2c13c736e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.198634 1166 tablet_replica.cc:331] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:04.199528 1165 tablet_bootstrap.cc:654] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.201500 1166 raft_consensus.cc:2238] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.202585 1160 ts_tablet_manager.cc:1918] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.202714 1160 log.cc:1198] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f1c5a6b9681d4ef6a36e308ab2916423
I20250109 20:53:04.202960 1158 tablet_bootstrap.cc:654] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.203029 1160 ts_tablet_manager.cc:1939] T f1c5a6b9681d4ef6a36e308ab2916423 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:04.203604 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1148) took 1125 ms. Trace:
I20250109 20:53:04.203572 1166 raft_consensus.cc:2267] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.207674 747 tablet_service.cc:1467] Processing CreateTablet for tablet 0075c64fe3164e5688ab5f2c13c736e6 (DEFAULT_TABLE table=default.Table_d4c1d0cf8d7f48bfbe31ec25d79cd5d2 [id=3551c51ff33b4789b8e5d733e5ac3d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.208781 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 0075c64fe3164e5688ab5f2c13c736e6 (DEFAULT_TABLE table=default.Table_d4c1d0cf8d7f48bfbe31ec25d79cd5d2 [id=3551c51ff33b4789b8e5d733e5ac3d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.209152 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0075c64fe3164e5688ab5f2c13c736e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.210830 756 tablet_service.cc:1467] Processing CreateTablet for tablet 8a22e629234a4456a63b60017767e67c (DEFAULT_TABLE table=default.Table_93fecf5906e3423684391032f8b408c8 [id=7d9429ececcf4c9593a1976cfb235c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.210126 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 7558fa96ea4e4fdd81b783880649d5a6 (DEFAULT_TABLE table=default.Table_cfdca0092aa74d3e805ce6c6132ca08f [id=a9eeed78abed436785c11691efd80269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.211472 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7558fa96ea4e4fdd81b783880649d5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.211688 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3d5e72e3aaea4360b7c8f619a831c1f1 (DEFAULT_TABLE table=default.Table_0ef997a787824972852640e0fa2bfebb [id=96f454e79051412b98fa7863c2175096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.212316 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 3d5e72e3aaea4360b7c8f619a831c1f1 (DEFAULT_TABLE table=default.Table_0ef997a787824972852640e0fa2bfebb [id=96f454e79051412b98fa7863c2175096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.212615 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5e72e3aaea4360b7c8f619a831c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.213105 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8a22e629234a4456a63b60017767e67c (DEFAULT_TABLE table=default.Table_93fecf5906e3423684391032f8b408c8 [id=7d9429ececcf4c9593a1976cfb235c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.213413 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a22e629234a4456a63b60017767e67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.214990 755 tablet_service.cc:1467] Processing CreateTablet for tablet e3e061d0e2c2432e9df424acee45f9bb (DEFAULT_TABLE table=default.Table_67e322e415e7458f88f98651dd786dfb [id=7153f6bd446f4837b792b61fc679db0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.215993 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 5964fcf72cc3430eb8083a8850d095a4 (DEFAULT_TABLE table=default.Table_1967d8c334354d17877612396cf2ce5a [id=04d2dea4a4704ab4bf16d2904e567237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.216301 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5964fcf72cc3430eb8083a8850d095a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.216691 751 tablet_service.cc:1467] Processing CreateTablet for tablet 5964fcf72cc3430eb8083a8850d095a4 (DEFAULT_TABLE table=default.Table_1967d8c334354d17877612396cf2ce5a [id=04d2dea4a4704ab4bf16d2904e567237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.216939 740 tablet_service.cc:1467] Processing CreateTablet for tablet bd6c36d0ccc0463b910c9abbe191b1bd (DEFAULT_TABLE table=default.Table_0446eb38ff3943c098a5979f8bc09fe2 [id=fabfa84cc544432c8d934007bac8b12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.217375 1016 tablet_service.cc:1467] Processing CreateTablet for tablet bd6c36d0ccc0463b910c9abbe191b1bd (DEFAULT_TABLE table=default.Table_0446eb38ff3943c098a5979f8bc09fe2 [id=fabfa84cc544432c8d934007bac8b12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.217665 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6c36d0ccc0463b910c9abbe191b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.220230 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a22e629234a4456a63b60017767e67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.209875 1166 ts_tablet_manager.cc:1905] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.223838 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5e72e3aaea4360b7c8f619a831c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.224817 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5964fcf72cc3430eb8083a8850d095a4 (DEFAULT_TABLE table=default.Table_1967d8c334354d17877612396cf2ce5a [id=04d2dea4a4704ab4bf16d2904e567237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.225134 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5964fcf72cc3430eb8083a8850d095a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.228091 1157 ts_tablet_manager.cc:1918] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.228181 1157 log.cc:1198] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/18f85944f66543fe8dd634ad0da0728d
I20250109 20:53:04.228420 1157 ts_tablet_manager.cc:1939] T 18f85944f66543fe8dd634ad0da0728d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.228910 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e061d0e2c2432e9df424acee45f9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.228964 888 tablet_service.cc:1467] Processing CreateTablet for tablet bd6c36d0ccc0463b910c9abbe191b1bd (DEFAULT_TABLE table=default.Table_0446eb38ff3943c098a5979f8bc09fe2 [id=fabfa84cc544432c8d934007bac8b12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.229202 750 tablet_service.cc:1467] Processing CreateTablet for tablet 7558fa96ea4e4fdd81b783880649d5a6 (DEFAULT_TABLE table=default.Table_cfdca0092aa74d3e805ce6c6132ca08f [id=a9eeed78abed436785c11691efd80269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.229281 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6c36d0ccc0463b910c9abbe191b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.229341 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6c36d0ccc0463b910c9abbe191b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.229696 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5964fcf72cc3430eb8083a8850d095a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.232954 881 tablet_service.cc:1467] Processing CreateTablet for tablet e3e061d0e2c2432e9df424acee45f9bb (DEFAULT_TABLE table=default.Table_67e322e415e7458f88f98651dd786dfb [id=7153f6bd446f4837b792b61fc679db0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.233245 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e061d0e2c2432e9df424acee45f9bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.233809 1110 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.DeleteTable -> {remote=127.0.104.62:37941, user_credentials={real_user=slave}} blocked reactor thread for 81201.1us
I20250109 20:53:04.237918 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0075c64fe3164e5688ab5f2c13c736e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.238287 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a07934849b084f54a2033b79378af24e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.238793 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a07934849b084f54a2033b79378af24e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.240136 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d0b623301b834635896597ef9d332c21" } new_table_name: "default.Table_9a9cbf596d89428194e0027fc7e52d1a" modify_external_catalogs: true
I20250109 20:53:04.241762 1157 tablet_replica.cc:331] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:04.241926 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_409e2ace4d204027a7d56be14d3f9c9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.242324 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_409e2ace4d204027a7d56be14d3f9c9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.242391 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 18f85944f66543fe8dd634ad0da0728d (table default.Table_9cbf7236e62e4c22a28e6ccb34f76495 [id=f3bf80bcfbb24031a4050f6f53569e93]) successfully deleted
I20250109 20:53:04.242789 1157 raft_consensus.cc:2238] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.243450 1157 raft_consensus.cc:2267] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.245090 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cae85363f473456ea6dff908e3978813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.245532 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cae85363f473456ea6dff908e3978813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.245602 883 tablet_service.cc:1467] Processing CreateTablet for tablet 3d5e72e3aaea4360b7c8f619a831c1f1 (DEFAULT_TABLE table=default.Table_0ef997a787824972852640e0fa2bfebb [id=96f454e79051412b98fa7863c2175096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.245949 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5e72e3aaea4360b7c8f619a831c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.246523 1157 ts_tablet_manager.cc:1905] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.247014 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_38af1d150640411a90cf87f30d618cb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.247066 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_776ff5756e184f24bcd08d48461ec4e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.247409 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38af1d150640411a90cf87f30d618cb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:04.247449 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_776ff5756e184f24bcd08d48461ec4e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.248898 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_201a598c2da7487bbb7cd29c9a0d9ae1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.249302 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_201a598c2da7487bbb7cd29c9a0d9ae1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.251006 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a34657904fd9418f8760104fbcc555e1" } new_table_name: "default.Table_3b8226ac2b1748b79200389ed44d6ec6" modify_external_catalogs: true
I20250109 20:53:04.251609 2368 consensus_queue.cc:1035] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:04.253011 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8a22e629234a4456a63b60017767e67c (DEFAULT_TABLE table=default.Table_93fecf5906e3423684391032f8b408c8 [id=7d9429ececcf4c9593a1976cfb235c6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.253198 2368 raft_consensus.cc:491] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.253365 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a22e629234a4456a63b60017767e67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.253340 2368 raft_consensus.cc:513] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.253901 2368 leader_election.cc:290] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.254364 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4870a667cd3e410590923c9e2497af54" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:04.254951 990 leader_election.cc:343] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.255326 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_10c76d34136a4a63afcc050df2123e97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.255610 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4870a667cd3e410590923c9e2497af54" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:04.255827 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10c76d34136a4a63afcc050df2123e97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:04.256300 989 leader_election.cc:343] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.256433 989 leader_election.cc:304] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.256865 2368 raft_consensus.cc:2743] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.260141 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7558fa96ea4e4fdd81b783880649d5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.262108 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cc87fbc188284d3f9099de392fb15757" } modify_external_catalogs: true
I20250109 20:53:04.262323 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cc87fbc188284d3f9099de392fb15757" } modify_external_catalogs: true
I20250109 20:53:04.264449 745 tablet_service.cc:1467] Processing CreateTablet for tablet 71db3cb472bd4534a2efe1918cec848f (DEFAULT_TABLE table=default.Table_d74b3ee84e164e75a624c43034693fc9 [id=6b4cf82c617e4e7092ff5ee9145382ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.264864 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71db3cb472bd4534a2efe1918cec848f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.267511 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1f012e6f7de14e748ed0497f6dfc2a00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.267987 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f012e6f7de14e748ed0497f6dfc2a00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:04.269083 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3997167fc6ee4cd0ae2401aa8924f684 consensus state has no leader
W20250109 20:53:04.270931 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c79ecf9db554b50b0a61325275b9bee consensus state has no leader
I20250109 20:53:04.273195 1158 tablet_bootstrap.cc:492] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:04.273322 1158 ts_tablet_manager.cc:1397] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250109 20:53:04.275391 1016 tablet_service.cc:1467] Processing CreateTablet for tablet e7e7f4dde1d641028863e5a9692bf282 (DEFAULT_TABLE table=default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.275856 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7e7f4dde1d641028863e5a9692bf282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.276949 1019 tablet_service.cc:1467] Processing CreateTablet for tablet e3e061d0e2c2432e9df424acee45f9bb (DEFAULT_TABLE table=default.Table_67e322e415e7458f88f98651dd786dfb [id=7153f6bd446f4837b792b61fc679db0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.277143 1158 raft_consensus.cc:357] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.277329 1158 raft_consensus.cc:383] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.277391 1158 raft_consensus.cc:738] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.278802 1158 consensus_queue.cc:260] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.277413 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e061d0e2c2432e9df424acee45f9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.280175 1158 ts_tablet_manager.cc:1428] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:53:04.281313 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_10f45fa0e77c4c79a77da8ec132dc497" } new_table_name: "default.Table_a499646769274f8aa06cd2d975344794" modify_external_catalogs: true
I20250109 20:53:04.281999 882 tablet_service.cc:1467] Processing CreateTablet for tablet e7e7f4dde1d641028863e5a9692bf282 (DEFAULT_TABLE table=default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.282485 1158 tablet_bootstrap.cc:492] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:04.284713 1158 tablet_bootstrap.cc:654] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.285267 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7e7f4dde1d641028863e5a9692bf282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.277364 1166 ts_tablet_manager.cc:1918] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:04.289196 893 tablet_service.cc:1467] Processing CreateTablet for tablet 7558fa96ea4e4fdd81b783880649d5a6 (DEFAULT_TABLE table=default.Table_cfdca0092aa74d3e805ce6c6132ca08f [id=a9eeed78abed436785c11691efd80269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.289563 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7558fa96ea4e4fdd81b783880649d5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.290320 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f5bea97ad34a41b1a53d274d1c918082" } new_table_name: "default.Table_c10042984e6447f7ae5aec3b79e7e0b9" modify_external_catalogs: true
I20250109 20:53:04.293064 1166 log.cc:1198] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/06e615ce8e124cfe9432778b6b58e117
I20250109 20:53:04.294551 1166 ts_tablet_manager.cc:1939] T 06e615ce8e124cfe9432778b6b58e117 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:04.297344 2368 raft_consensus.cc:491] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.284834 746 tablet_service.cc:1467] Processing CreateTablet for tablet e7e7f4dde1d641028863e5a9692bf282 (DEFAULT_TABLE table=default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.297504 2368 raft_consensus.cc:513] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.297747 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7e7f4dde1d641028863e5a9692bf282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.297691 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f0b3bb838274407daa9df9dc44897208"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.298153 2368 leader_election.cc:290] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:53:04.298182 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0b3bb838274407daa9df9dc44897208 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.298857 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048be84b36a1450a818f05fb36cbf3b9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.291381 2413 consensus_queue.cc:1035] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:04.299214 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 06e615ce8e124cfe9432778b6b58e117 (table default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]) successfully deleted
W20250109 20:53:04.299332 990 leader_election.cc:343] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.300084 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048be84b36a1450a818f05fb36cbf3b9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:04.300661 989 leader_election.cc:343] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.300791 989 leader_election.cc:304] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:04.300810 712 consensus_peers.cc:487] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:04.301034 1166 tablet_replica.cc:331] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:04.301471 2368 raft_consensus.cc:2743] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.303771 1166 raft_consensus.cc:2238] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.306221 1166 raft_consensus.cc:2267] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.308646 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bd642e5ec83a4da88eaa2eea3ffca362
I20250109 20:53:04.309495 2368 raft_consensus.cc:491] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.309525 890 tablet_service.cc:1514] Processing DeleteTablet for tablet bd642e5ec83a4da88eaa2eea3ffca362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.309640 2368 raft_consensus.cc:513] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.309955 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet bd642e5ec83a4da88eaa2eea3ffca362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.310269 2368 leader_election.cc:290] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.310971 890 tablet_service.cc:1467] Processing CreateTablet for tablet cc3108fac86b4b15a1d9ab0fe84f05b6 (DEFAULT_TABLE table=default.Table_1e89724f79a84156954c1455cfa74676 [id=3809fef620b946ed8552e66c2a1a1c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.311131 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8ea5ea7fa44866b6d7160466f5a896" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.311431 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc3108fac86b4b15a1d9ab0fe84f05b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.311612 990 leader_election.cc:343] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.312152 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8ea5ea7fa44866b6d7160466f5a896" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:04.312624 989 leader_election.cc:343] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.312742 989 leader_election.cc:304] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.313114 2368 raft_consensus.cc:2743] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.313158 1166 ts_tablet_manager.cc:1905] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.317414 1165 tablet_bootstrap.cc:492] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:04.318287 747 tablet_service.cc:1514] Processing DeleteTablet for tablet bd642e5ec83a4da88eaa2eea3ffca362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:04.318754 1165 ts_tablet_manager.cc:1397] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.000s
I20250109 20:53:04.319268 747 tablet_service.cc:1467] Processing CreateTablet for tablet cc3108fac86b4b15a1d9ab0fe84f05b6 (DEFAULT_TABLE table=default.Table_1e89724f79a84156954c1455cfa74676 [id=3809fef620b946ed8552e66c2a1a1c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.319597 1165 raft_consensus.cc:357] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.320645 2413 consensus_queue.cc:1035] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:04.321066 1176 tablet_bootstrap.cc:492] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:04.321197 1176 ts_tablet_manager.cc:1397] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.201s user 0.006s sys 0.015s
I20250109 20:53:04.321885 750 tablet_service.cc:1467] Processing CreateTablet for tablet 83a1bbe748464149b3d2696e6798bac0 (DEFAULT_TABLE table=default.Table_4506c544ae5f4b2e8b2883e0e5105ddc [id=937b2824bb2e48e589c88bbae1026962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.313817 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 71db3cb472bd4534a2efe1918cec848f (DEFAULT_TABLE table=default.Table_d74b3ee84e164e75a624c43034693fc9 [id=6b4cf82c617e4e7092ff5ee9145382ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.322424 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71db3cb472bd4534a2efe1918cec848f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.325165 1165 raft_consensus.cc:383] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.325415 1165 raft_consensus.cc:738] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.325870 1165 consensus_queue.cc:260] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.326959 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 83a1bbe748464149b3d2696e6798bac0 (DEFAULT_TABLE table=default.Table_4506c544ae5f4b2e8b2883e0e5105ddc [id=937b2824bb2e48e589c88bbae1026962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.327885 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a1bbe748464149b3d2696e6798bac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.328254 1020 tablet_service.cc:1467] Processing CreateTablet for tablet cc3108fac86b4b15a1d9ab0fe84f05b6 (DEFAULT_TABLE table=default.Table_1e89724f79a84156954c1455cfa74676 [id=3809fef620b946ed8552e66c2a1a1c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.328599 1016 tablet_service.cc:1467] Processing CreateTablet for tablet c4cff74250cd435e8a52c010c11eff9b (DEFAULT_TABLE table=default.Table_8ec506c30a204667919a43f12ab10a09 [id=b3e8917acae141ee970adcd2425b958e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.328609 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc3108fac86b4b15a1d9ab0fe84f05b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.329382 1165 ts_tablet_manager.cc:1428] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250109 20:53:04.203696 1160 rpcz_store.cc:276] 0109 20:53:03.077718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.077757 (+ 39us) service_pool.cc:225] Handling call
0109 20:53:03.995541 (+917784us) ts_tablet_manager.cc:1145] Deleting tablet f1c5a6b9681d4ef6a36e308ab2916423
0109 20:53:03.995551 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.098114 (+102563us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.202522 (+104408us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.203587 (+ 1065us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":917620}
I20250109 20:53:04.324317 890 tablet_service.cc:1467] Processing CreateTablet for tablet 83a1bbe748464149b3d2696e6798bac0 (DEFAULT_TABLE table=default.Table_4506c544ae5f4b2e8b2883e0e5105ddc [id=937b2824bb2e48e589c88bbae1026962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.332034 881 tablet_service.cc:1467] Processing CreateTablet for tablet c4cff74250cd435e8a52c010c11eff9b (DEFAULT_TABLE table=default.Table_8ec506c30a204667919a43f12ab10a09 [id=b3e8917acae141ee970adcd2425b958e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.333729 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f1c5a6b9681d4ef6a36e308ab2916423 (table default.Table_68ffdf1aeefe420ea24aa1102012f77c [id=4d826d307bff4ab2b43a9d8b718055d6]) successfully deleted
I20250109 20:53:04.334944 1176 raft_consensus.cc:357] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.335732 1165 tablet_bootstrap.cc:492] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:04.336704 1176 raft_consensus.cc:383] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.337056 1176 raft_consensus.cc:738] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.337502 1176 consensus_queue.cc:260] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.338549 1176 ts_tablet_manager.cc:1428] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250109 20:53:04.339042 751 tablet_service.cc:1467] Processing CreateTablet for tablet c4cff74250cd435e8a52c010c11eff9b (DEFAULT_TABLE table=default.Table_8ec506c30a204667919a43f12ab10a09 [id=b3e8917acae141ee970adcd2425b958e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.339349 1176 tablet_bootstrap.cc:492] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:04.340030 740 tablet_service.cc:1467] Processing CreateTablet for tablet ab7c9787bca04835bd9ddce94bd54ce4 (DEFAULT_TABLE table=default.Table_4511b714cc7844378468fd7bd37c0025 [id=443402a33fa740f9997a7bd9a61e5345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.340827 1176 tablet_bootstrap.cc:654] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:04.340883 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04b4247f52ce4d33bf49d9625a1592bf consensus state has no leader
W20250109 20:53:04.342635 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e67d92bb9034dfaacdca9cbcea57550 consensus state has no leader
I20250109 20:53:04.347687 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]). Expected version 1 got 0
I20250109 20:53:04.349097 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_09f1cb68ab054ac4836cd848a45b11de" } new_table_name: "default.Table_88165ba86a94484a979e32aca59503dd" modify_external_catalogs: true
I20250109 20:53:04.352409 755 tablet_service.cc:1467] Processing CreateTablet for tablet 88a31e2d5bc94f82ae1cbdf6cac5d77c (DEFAULT_TABLE table=default.Table_a0874353369a4986b5b3e814940a97a0 [id=5f5747ad16774467802ac3ab3bab3d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.353684 742 tablet_service.cc:1467] Processing CreateTablet for tablet ba14bef371c444ab9d84a42137a01113 (DEFAULT_TABLE table=default.Table_ff364e95ed60447ca22919c7fba6e242 [id=1926b021b38b4b00a7c92a30bef67553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.354024 756 tablet_service.cc:1467] Processing CreateTablet for tablet 1b895b0c2ba2445583d54b4f515ea7d5 (DEFAULT_TABLE table=default.Table_56a50e008101433aba64aa57711f3688 [id=177fa8177a8f41cc8d4e68ef29371d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.354959 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4cff74250cd435e8a52c010c11eff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.355118 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc3108fac86b4b15a1d9ab0fe84f05b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.355309 754 tablet_service.cc:1467] Processing CreateTablet for tablet 34188f7707704c398d6b86a1a7cdc3b3 (DEFAULT_TABLE table=default.Table_69a8a32a033547b28e591ec3684a668e [id=073c19edccb34dc59798cc248be4aa31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.355407 1157 ts_tablet_manager.cc:1918] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.355501 1157 log.cc:1198] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ea477dc845614ea78e71e45f862b57ff
I20250109 20:53:04.355525 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b895b0c2ba2445583d54b4f515ea7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.355787 1157 ts_tablet_manager.cc:1939] T ea477dc845614ea78e71e45f862b57ff P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.355928 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba14bef371c444ab9d84a42137a01113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.356437 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a31e2d5bc94f82ae1cbdf6cac5d77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.356876 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7c9787bca04835bd9ddce94bd54ce4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.357801 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 04b4247f52ce4d33bf49d9625a1592bf (table default.Table_a499646769274f8aa06cd2d975344794 [id=412d889629a84f50aca3788e40f2916e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 04b4247f52ce4d33bf49d9625a1592bf consensus state has no leader
I20250109 20:53:04.362439 451 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:53:04.335777 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4cff74250cd435e8a52c010c11eff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.367475 2475 raft_consensus.cc:491] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.367630 2475 raft_consensus.cc:513] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.368336 2475 leader_election.cc:290] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.368600 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce062dc0ac144a19f8fa3eef4a099b4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:04.370263 451 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]). Expected version 1 got 0
W20250109 20:53:04.370940 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 51918.2us
I20250109 20:53:04.373967 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34188f7707704c398d6b86a1a7cdc3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.374536 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ea477dc845614ea78e71e45f862b57ff (table default.Table_1b9aa886edd542fc8039fd921d1048c8 [id=d64f7249495e433fa9c57277ee5eaba9]) successfully deleted
I20250109 20:53:04.374627 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a1bbe748464149b3d2696e6798bac0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.374900 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5f953f43877c4a738ffd8255afbf8275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f953f43877c4a738ffd8255afbf8275 already in progress: creating tablet
W20250109 20:53:04.375188 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3293f49190684b72a5c41031e621bd36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3293f49190684b72a5c41031e621bd36 already in progress: creating tablet
I20250109 20:53:04.375509 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a1bbe748464149b3d2696e6798bac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.341982 1030 tablet_service.cc:1467] Processing CreateTablet for tablet ab7c9787bca04835bd9ddce94bd54ce4 (DEFAULT_TABLE table=default.Table_4511b714cc7844378468fd7bd37c0025 [id=443402a33fa740f9997a7bd9a61e5345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.356503 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 88a31e2d5bc94f82ae1cbdf6cac5d77c (DEFAULT_TABLE table=default.Table_a0874353369a4986b5b3e814940a97a0 [id=5f5747ad16774467802ac3ab3bab3d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.356631 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ba14bef371c444ab9d84a42137a01113 (DEFAULT_TABLE table=default.Table_ff364e95ed60447ca22919c7fba6e242 [id=1926b021b38b4b00a7c92a30bef67553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.356992 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 1b895b0c2ba2445583d54b4f515ea7d5 (DEFAULT_TABLE table=default.Table_56a50e008101433aba64aa57711f3688 [id=177fa8177a8f41cc8d4e68ef29371d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.335960 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1149) took 1242 ms. Trace:
I20250109 20:53:04.343690 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 9558dfe563d94482bf496e19b8a80edc with delete_type TABLET_DATA_DELETED (replaced by tablet a8ad256c6bf64a699ede8ff3cee5dac1) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.352326 2032 raft_consensus.cc:491] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.380695 1160 rpcz_store.cc:276] 0109 20:53:03.093869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.093946 (+ 77us) service_pool.cc:225] Handling call
0109 20:53:04.329556 (+1235610us) ts_tablet_manager.cc:1145] Deleting tablet 17940f311b0440a59a42cc57f3a0bc4c
0109 20:53:04.335839 (+ 6283us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.335946 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1235426}
I20250109 20:53:04.380888 2032 raft_consensus.cc:513] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.381207 1160 tablet_replica.cc:331] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
W20250109 20:53:04.381289 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 17940f311b0440a59a42cc57f3a0bc4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17940f311b0440a59a42cc57f3a0bc4c already in progress: creating tablet
I20250109 20:53:04.381397 1160 raft_consensus.cc:2238] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.381525 2032 leader_election.cc:290] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:04.353395 885 tablet_service.cc:1467] Processing CreateTablet for tablet 88a31e2d5bc94f82ae1cbdf6cac5d77c (DEFAULT_TABLE table=default.Table_a0874353369a4986b5b3e814940a97a0 [id=5f5747ad16774467802ac3ab3bab3d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.382378 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a31e2d5bc94f82ae1cbdf6cac5d77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.383364 1165 tablet_bootstrap.cc:654] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.353433 895 tablet_service.cc:1467] Processing CreateTablet for tablet ba14bef371c444ab9d84a42137a01113 (DEFAULT_TABLE table=default.Table_ff364e95ed60447ca22919c7fba6e242 [id=1926b021b38b4b00a7c92a30bef67553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.384989 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba14bef371c444ab9d84a42137a01113. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.386734 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55bf7303e7354aa99a7b5bb489f442fe consensus state has no leader
I20250109 20:53:04.354571 886 tablet_service.cc:1467] Processing CreateTablet for tablet 1b895b0c2ba2445583d54b4f515ea7d5 (DEFAULT_TABLE table=default.Table_56a50e008101433aba64aa57711f3688 [id=177fa8177a8f41cc8d4e68ef29371d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.354941 892 tablet_service.cc:1467] Processing CreateTablet for tablet 34188f7707704c398d6b86a1a7cdc3b3 (DEFAULT_TABLE table=default.Table_69a8a32a033547b28e591ec3684a668e [id=073c19edccb34dc59798cc248be4aa31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.388365 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b895b0c2ba2445583d54b4f515ea7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.388557 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34188f7707704c398d6b86a1a7cdc3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.392755 739 tablet_service.cc:1467] Processing CreateTablet for tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7 (DEFAULT_TABLE table=default.Table_aa3f8eb194bf4a798c9efffc02e16f48 [id=f46bf4baee9643d8813926e3af2900de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.394040 882 tablet_service.cc:1467] Processing CreateTablet for tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7 (DEFAULT_TABLE table=default.Table_aa3f8eb194bf4a798c9efffc02e16f48 [id=f46bf4baee9643d8813926e3af2900de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.394512 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.398015 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 801838c21f70424ba1b2d89585bda643 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:04.400436 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4cff74250cd435e8a52c010c11eff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.402179 1017 tablet_service.cc:1467] Processing CreateTablet for tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7 (DEFAULT_TABLE table=default.Table_aa3f8eb194bf4a798c9efffc02e16f48 [id=f46bf4baee9643d8813926e3af2900de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.402639 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 34188f7707704c398d6b86a1a7cdc3b3 (DEFAULT_TABLE table=default.Table_69a8a32a033547b28e591ec3684a668e [id=073c19edccb34dc59798cc248be4aa31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.402879 1166 ts_tablet_manager.cc:1918] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.403021 1166 log.cc:1198] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/5f953f43877c4a738ffd8255afbf8275
I20250109 20:53:04.403434 1166 ts_tablet_manager.cc:1939] T 5f953f43877c4a738ffd8255afbf8275 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:04.403694 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet dc235614171f4a61b75492ddeb3f0fe6 with delete_type TABLET_DATA_DELETED (replaced by tablet 67be9533983443cbb8fc52031216a973) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:04.404258 989 leader_election.cc:343] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.408372 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a31e2d5bc94f82ae1cbdf6cac5d77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.411221 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba14bef371c444ab9d84a42137a01113. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.411376 989 consensus_peers.cc:487] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:04.411864 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7c9787bca04835bd9ddce94bd54ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.381547 1160 raft_consensus.cc:2267] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.415628 1160 ts_tablet_manager.cc:1905] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.416055 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b895b0c2ba2445583d54b4f515ea7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.416455 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34188f7707704c398d6b86a1a7cdc3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.417168 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.418359 1166 tablet_replica.cc:331] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:04.418557 1166 raft_consensus.cc:2238] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.418694 1166 raft_consensus.cc:2267] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.419234 1166 ts_tablet_manager.cc:1905] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.419366 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 5f953f43877c4a738ffd8255afbf8275 (table default.Table_3254b432932b4d9bb83cb8a2f26e5514 [id=fa8fa63dbd44442b81a3b9640d42680e]) successfully deleted
W20250109 20:53:04.419601 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3293f49190684b72a5c41031e621bd36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3293f49190684b72a5c41031e621bd36 already in progress: creating tablet
I20250109 20:53:04.421466 1165 tablet_bootstrap.cc:492] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:04.421790 1165 ts_tablet_manager.cc:1397] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.086s user 0.002s sys 0.002s
I20250109 20:53:04.422641 1165 raft_consensus.cc:357] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.422957 1165 raft_consensus.cc:383] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.423079 1165 raft_consensus.cc:738] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.423395 1165 consensus_queue.cc:260] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.423975 1165 ts_tablet_manager.cc:1428] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:04.426666 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5b523f187b4bcfb69fbaf7e4ecc9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.429404 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0538f4d3608a454b9dd21a0fa42068a0" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.435523 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce062dc0ac144a19f8fa3eef4a099b4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.436853 1165 tablet_bootstrap.cc:492] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:04.437191 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8aa7a8f99b5a4c5293df93fcbd7ba167" } modify_external_catalogs: true
I20250109 20:53:04.437533 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:04.438832 1165 tablet_bootstrap.cc:654] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.443584 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e778e2d95d0240ef965b87c9f44fc0ba" } modify_external_catalogs: true
I20250109 20:53:04.443917 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6ba9903051db4a88adae464061f959d8" } modify_external_catalogs: true
I20250109 20:53:04.444198 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_002027b48ee14395a6b054a17005d2ab" } modify_external_catalogs: true
I20250109 20:53:04.452152 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 801838c21f70424ba1b2d89585bda643
I20250109 20:53:04.452644 446 catalog_manager.cc:6596] ReplaceTablet: tablet 801838c21f70424ba1b2d89585bda643 deleted and replaced by tablet f12b13cead8942e2a4e874dde17a0802
I20250109 20:53:04.453214 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 801838c21f70424ba1b2d89585bda643 with delete_type TABLET_DATA_DELETED (replaced by tablet f12b13cead8942e2a4e874dde17a0802) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.454540 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_002027b48ee14395a6b054a17005d2ab" } modify_external_catalogs: true
I20250109 20:53:04.457533 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6ba9903051db4a88adae464061f959d8" } modify_external_catalogs: true
I20250109 20:53:04.458326 1165 tablet_bootstrap.cc:492] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:04.458462 1165 ts_tablet_manager.cc:1397] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250109 20:53:04.459156 1165 raft_consensus.cc:357] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.459299 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e778e2d95d0240ef965b87c9f44fc0ba" } modify_external_catalogs: true
I20250109 20:53:04.460476 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8aa7a8f99b5a4c5293df93fcbd7ba167" } modify_external_catalogs: true
I20250109 20:53:04.444578 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_75c0dbbb51764601811f6bff5ca923c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.461215 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75c0dbbb51764601811f6bff5ca923c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:04.464605 989 consensus_peers.cc:487] T 4f8419d8cc7e475ea43dd263cae87aca P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:04.464957 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a2f187de26b34ff2a1d1be7289628fcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.465449 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2f187de26b34ff2a1d1be7289628fcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:04.465816 2223 log.cc:926] Time spent T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250109 20:53:04.467406 1165 raft_consensus.cc:383] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.467499 1165 raft_consensus.cc:738] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.467710 1165 consensus_queue.cc:260] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.468051 883 tablet_service.cc:1467] Processing CreateTablet for tablet ab7c9787bca04835bd9ddce94bd54ce4 (DEFAULT_TABLE table=default.Table_4511b714cc7844378468fd7bd37c0025 [id=443402a33fa740f9997a7bd9a61e5345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.468447 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7c9787bca04835bd9ddce94bd54ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.473857 1176 tablet_bootstrap.cc:492] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:04.474042 1176 ts_tablet_manager.cc:1397] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.004s
I20250109 20:53:04.474926 1165 ts_tablet_manager.cc:1428] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250109 20:53:04.474961 2470 raft_consensus.cc:491] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.475230 1165 tablet_bootstrap.cc:492] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:04.475219 2470 raft_consensus.cc:513] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.476812 1165 tablet_bootstrap.cc:654] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.477945 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cbf320b748bb4162a3ce1ef93588df7b" } new_table_name: "default.Table_02829bc5be564d358bcca8febc5c3bd5" modify_external_catalogs: true
I20250109 20:53:04.479499 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bc513c9e60954461ad69deebb17af802"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.480108 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc513c9e60954461ad69deebb17af802 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.480103 1176 raft_consensus.cc:357] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.480334 1176 raft_consensus.cc:383] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.480419 1176 raft_consensus.cc:738] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.480661 1176 consensus_queue.cc:260] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.481632 1176 ts_tablet_manager.cc:1428] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:53:04.482174 1176 tablet_bootstrap.cc:492] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:04.482204 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_43180160d5a044548b7cf16b6c122bfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.482721 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43180160d5a044548b7cf16b6c122bfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.485787 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c91107370a3c405c8532e0931e7440c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.486207 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c91107370a3c405c8532e0931e7440c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.487573 882 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd3a1b1d3e44feda6dbaa8556e377fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.487628 2032 raft_consensus.cc:491] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.487758 2032 raft_consensus.cc:513] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.488577 2032 leader_election.cc:290] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:04.489626 1157 tablet_replica.cc:331] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:04.489825 1157 raft_consensus.cc:2238] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.489969 1157 raft_consensus.cc:2267] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.490626 1157 ts_tablet_manager.cc:1905] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.491626 2469 raft_consensus.cc:491] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.494668 919 raft_consensus.cc:3054] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:04.497200 854 leader_election.cc:343] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 16c9f7274a7542e89a19bd4afcf89205
I20250109 20:53:04.497721 2032 raft_consensus.cc:491] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.497908 2032 raft_consensus.cc:513] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.498548 2032 leader_election.cc:290] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:04.499370 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d0dc4addb7fd4d23822b25af5e288118
I20250109 20:53:04.491750 2469 raft_consensus.cc:513] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.502997 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d409409813420689134773c627833b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.502897 882 tablet_service.cc:1514] Processing DeleteTablet for tablet d0dc4addb7fd4d23822b25af5e288118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:04.503937 854 leader_election.cc:343] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 93d409409813420689134773c627833b
I20250109 20:53:04.516294 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a9aba1c7b424db21c30b2e92323c9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:04.516031 1160 ts_tablet_manager.cc:1918] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:53:04.517146 712 leader_election.cc:343] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.517349 1160 log.cc:1198] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/219a4b508050490cb3f1ee418e0edbf8
I20250109 20:53:04.518975 2469 leader_election.cc:290] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:04.520175 1160 ts_tablet_manager.cc:1939] T 219a4b508050490cb3f1ee418e0edbf8 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:04.523840 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 75ad55b91a6c405aae3ed456a8c4d9a3
I20250109 20:53:04.524521 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_809a4f599c56477ca32143d77d2788fc" } new_table_name: "default.Table_8b8f55998d744172960e227caea20dcc" modify_external_catalogs: true
I20250109 20:53:04.526381 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b087b7546a39451089666a36d029e645
I20250109 20:53:04.526619 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1152) took 1292 ms. Trace:
I20250109 20:53:04.527220 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0810c66546f749f4bde30df12a982a14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.527698 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:04.527813 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0810c66546f749f4bde30df12a982a14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.527324 1160 rpcz_store.cc:276] 0109 20:53:03.234343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.234904 (+ 561us) service_pool.cc:225] Handling call
0109 20:53:04.381139 (+1146235us) ts_tablet_manager.cc:1145] Deleting tablet 219a4b508050490cb3f1ee418e0edbf8
0109 20:53:04.381148 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.488580 (+107432us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.515006 (+ 26426us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.526328 (+ 11322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146025}
I20250109 20:53:04.529697 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 219a4b508050490cb3f1ee418e0edbf8 (table default.Table_8d504623c1894cb8843ead0ee687ea13 [id=5c7e66cff6b14f9a81029cea36224460]) successfully deleted
I20250109 20:53:04.529870 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b087b7546a39451089666a36d029e645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.530107 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c6bc95b32aa04e25bbde02bf5e8fe4d9
I20250109 20:53:04.530694 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_75ec2484861c48b2afc25351a61ba546"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.531113 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75ec2484861c48b2afc25351a61ba546 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.531296 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1151) took 1296 ms. Trace:
I20250109 20:53:04.531888 1160 rpcz_store.cc:276] 0109 20:53:03.234223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.235908 (+ 1685us) service_pool.cc:225] Handling call
0109 20:53:04.530271 (+1294363us) ts_tablet_manager.cc:1145] Deleting tablet e0f47f9b1f8a479a9760526a4873af57
0109 20:53:04.530434 (+ 163us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.530909 (+ 475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293938}
I20250109 20:53:04.533022 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1f0817571af249be925827a9c80c6ffd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.533520 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f0817571af249be925827a9c80c6ffd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.534740 1157 ts_tablet_manager.cc:1918] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.534910 1157 log.cc:1198] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/9c3af2a00ab04b41915cffc3eb9206bc
I20250109 20:53:04.535171 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_70b2bddd5393467f8ae2ff2ea7d1a79d" } modify_external_catalogs: true
I20250109 20:53:04.535331 1157 ts_tablet_manager.cc:1939] T 9c3af2a00ab04b41915cffc3eb9206bc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.535351 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_70b2bddd5393467f8ae2ff2ea7d1a79d" } modify_external_catalogs: true
I20250109 20:53:04.535907 1157 tablet_replica.cc:331] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:53:04.536058 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e0f47f9b1f8a479a9760526a4873af57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f47f9b1f8a479a9760526a4873af57 already in progress: creating tablet
I20250109 20:53:04.536166 1157 raft_consensus.cc:2238] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.536325 1157 raft_consensus.cc:2267] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.536146 1160 tablet_replica.cc:331] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:04.536609 882 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.538318 1158 tablet_bootstrap.cc:492] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:04.537969 1160 raft_consensus.cc:2238] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.539525 1160 raft_consensus.cc:2267] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.542090 1160 ts_tablet_manager.cc:1905] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.544534 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:04.544844 1157 ts_tablet_manager.cc:1905] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.548767 1158 ts_tablet_manager.cc:1397] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.266s user 0.003s sys 0.005s
W20250109 20:53:04.556368 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 622cc5f673694546a594f968621f1011 consensus state has no leader
I20250109 20:53:04.556893 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1f2b650f7699459eadc01979a7bc8b90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.557020 1158 raft_consensus.cc:357] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.557188 1158 raft_consensus.cc:383] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.557237 1158 raft_consensus.cc:738] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:04.557438 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f2b650f7699459eadc01979a7bc8b90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.557413 1158 consensus_queue.cc:260] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.557858 1158 ts_tablet_manager.cc:1428] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250109 20:53:04.558097 1158 tablet_bootstrap.cc:492] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:04.559584 1158 tablet_bootstrap.cc:654] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:04.560590 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 already in progress: creating tablet
I20250109 20:53:04.503430 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dc2216d87ffe4dc288a86a9b772341d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.562779 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc2216d87ffe4dc288a86a9b772341d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.576315 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:04.578047 1160 ts_tablet_manager.cc:1918] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.578802 1160 log.cc:1198] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/0b0450d88e3e4ccaae4e5157ce5c80c1
I20250109 20:53:04.579702 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ebdd59dd13424ecf93c997fa5919477d" } new_table_name: "default.Table_c0ff6d3bb2624e6f9e386a0a38239d27" modify_external_catalogs: true
I20250109 20:53:04.580601 1160 ts_tablet_manager.cc:1939] T 0b0450d88e3e4ccaae4e5157ce5c80c1 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:04.581152 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8c8b587893cb4890887373eb89950b96" } new_table_name: "default.Table_bc7963618a2e45d6999e681eec0db896" modify_external_catalogs: true
W20250109 20:53:04.581921 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 108269us
I20250109 20:53:04.582322 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1153) took 1327 ms. Trace:
W20250109 20:53:04.583192 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ddc83d535408457fa9493592f527f8a4 consensus state has no leader
I20250109 20:53:04.583288 915 raft_consensus.cc:3054] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:04.586030 1157 ts_tablet_manager.cc:1918] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.591085 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8d34d92047e245fe92fe56a0e07f3b95" } new_table_name: "default.Table_16df12cac132428ca8cc7cac50d05578" modify_external_catalogs: true
I20250109 20:53:04.582767 1160 rpcz_store.cc:276] 0109 20:53:03.254792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.254852 (+ 60us) service_pool.cc:225] Handling call
0109 20:53:04.535459 (+1280607us) ts_tablet_manager.cc:1145] Deleting tablet 0b0450d88e3e4ccaae4e5157ce5c80c1
0109 20:53:04.535632 (+ 173us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.555096 (+ 19464us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.577448 (+ 22352us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.582136 (+ 4688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280314}
I20250109 20:53:04.592278 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 0b0450d88e3e4ccaae4e5157ce5c80c1 (table default.Table_70dd3dbabbbc4d52ab5ac6beda01d27d [id=234ab45ef1594b61b32024026c9f3318]) successfully deleted
I20250109 20:53:04.592201 1160 tablet_replica.cc:331] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:04.592845 1157 log.cc:1198] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/e090f72eb1e046319b825ee5fe126332
I20250109 20:53:04.593441 1157 ts_tablet_manager.cc:1939] T e090f72eb1e046319b825ee5fe126332 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.593962 2488 raft_consensus.cc:491] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.593539 1160 raft_consensus.cc:2238] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.594144 2488 raft_consensus.cc:513] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.595157 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f420b7e623243e093d26c95276c5af9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:04.595055 1160 raft_consensus.cc:2267] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:04.595788 712 leader_election.cc:343] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.596572 2488 leader_election.cc:290] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:04.597775 1157 tablet_replica.cc:331] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:04.599032 1160 ts_tablet_manager.cc:1905] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.599617 1157 raft_consensus.cc:2238] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.600023 1157 raft_consensus.cc:2267] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.600672 1157 ts_tablet_manager.cc:1905] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.601531 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a9aba1c7b424db21c30b2e92323c9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:04.603933 2475 raft_consensus.cc:491] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.604192 1176 tablet_bootstrap.cc:654] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:04.604326 990 leader_election.cc:343] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.605360 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet d0dc4addb7fd4d23822b25af5e288118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.606738 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.606992 1055 raft_consensus.cc:2463] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:04.606760 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.607560 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet b087b7546a39451089666a36d029e645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.608481 990 leader_election.cc:304] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.608999 2475 raft_consensus.cc:513] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.611742 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019a01afbcc452a9a9374c92dc131f3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:04.611806 713 leader_election.cc:304] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.611804 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404ee892ab79427baaef6d19c57699d0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:04.612082 2488 raft_consensus.cc:2798] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250109 20:53:04.612187 713 leader_election.cc:343] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8f420b7e623243e093d26c95276c5af9
I20250109 20:53:04.612211 2488 raft_consensus.cc:491] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:04.612449 2488 raft_consensus.cc:3054] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:04.612298 713 leader_election.cc:304] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.613091 2469 raft_consensus.cc:2743] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.613463 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 801838c21f70424ba1b2d89585bda643 with delete_type TABLET_DATA_DELETED (replaced by tablet f12b13cead8942e2a4e874dde17a0802) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.613875 2478 raft_consensus.cc:2743] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:04.614414 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 67886.7us
I20250109 20:53:04.615049 2488 raft_consensus.cc:513] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.615314 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019a01afbcc452a9a9374c92dc131f3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.615617 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404ee892ab79427baaef6d19c57699d0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.615845 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a9aba1c7b424db21c30b2e92323c9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:04.616271 712 leader_election.cc:343] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.617125 2475 leader_election.cc:290] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.616755 2470 leader_election.cc:290] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:53:04.618023 989 leader_election.cc:343] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.617883 2368 consensus_queue.cc:1035] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250109 20:53:04.618894 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0538f4d3608a454b9dd21a0fa42068a0" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
W20250109 20:53:04.621487 990 leader_election.cc:343] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:04.624528 989 leader_election.cc:343] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.625737 990 leader_election.cc:304] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.625808 2488 leader_election.cc:290] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:04.626118 990 leader_election.cc:343] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:04.626509 854 leader_election.cc:343] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 16c9f7274a7542e89a19bd4afcf89205
I20250109 20:53:04.627208 854 leader_election.cc:304] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.627718 2032 raft_consensus.cc:2743] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.628111 2368 raft_consensus.cc:2743] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.626209 990 leader_election.cc:304] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:04.628290 854 leader_election.cc:343] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 93d409409813420689134773c627833b
I20250109 20:53:04.628397 854 leader_election.cc:304] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.628759 2032 raft_consensus.cc:2743] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.629491 2368 raft_consensus.cc:2743] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.630300 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6a9aba1c7b424db21c30b2e92323c9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:04.630478 1052 raft_consensus.cc:3054] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:04.630880 2469 raft_consensus.cc:491] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.630995 2469 raft_consensus.cc:513] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.631459 2469 leader_election.cc:290] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:04.632092 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225046d45c1c43b488babf5da8de75d7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:04.630105 854 leader_election.cc:304] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:04.632778 712 leader_election.cc:343] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.633643 1166 ts_tablet_manager.cc:1918] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.633770 1166 log.cc:1198] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b74a4c1bdd8f4f6e9592ee5d97c58cc6
I20250109 20:53:04.634019 1166 ts_tablet_manager.cc:1939] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:04.637076 2470 raft_consensus.cc:491] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.637200 2470 raft_consensus.cc:513] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.637743 2470 leader_election.cc:290] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.638058 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea16fcc30f2c4b2eac53f20fcebd043f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:04.638777 990 leader_election.cc:343] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.639918 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea16fcc30f2c4b2eac53f20fcebd043f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:04.640069 1052 raft_consensus.cc:2463] T 8e6a9aba1c7b424db21c30b2e92323c9 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
W20250109 20:53:04.643208 989 leader_election.cc:343] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.643306 989 leader_election.cc:304] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.644394 2470 raft_consensus.cc:2743] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.650192 1157 ts_tablet_manager.cc:1918] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.650295 1157 log.cc:1198] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/80da78917f0b4931ab1705371c1fe5dd
I20250109 20:53:04.650610 1157 ts_tablet_manager.cc:1939] T 80da78917f0b4931ab1705371c1fe5dd P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.651968 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 (table default.Table_7ab416cb67944fa5b70140a8b8ab5e45 [id=5f7ff88269fa48ec80fe384008ba5780]) successfully deleted
W20250109 20:53:04.652644 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d2c8a6555a8e4228a9a80b15c6880c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c8a6555a8e4228a9a80b15c6880c84 already in progress: creating tablet
I20250109 20:53:04.652993 1166 tablet_replica.cc:331] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:04.653215 1166 raft_consensus.cc:2238] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.656147 1158 tablet_bootstrap.cc:492] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:04.656162 1166 raft_consensus.cc:2267] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.656637 1166 ts_tablet_manager.cc:1905] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.659605 1158 ts_tablet_manager.cc:1397] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.000s
I20250109 20:53:04.661909 1158 raft_consensus.cc:357] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.653455 1176 tablet_bootstrap.cc:492] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:04.663524 1158 raft_consensus.cc:383] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.663547 1176 ts_tablet_manager.cc:1397] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.182s user 0.004s sys 0.008s
I20250109 20:53:04.663661 1158 raft_consensus.cc:738] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.662286 1157 tablet_replica.cc:331] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:04.664019 1157 raft_consensus.cc:2238] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.664214 1157 raft_consensus.cc:2267] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.664207 1176 raft_consensus.cc:357] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.664388 1176 raft_consensus.cc:383] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.664441 1176 raft_consensus.cc:738] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.664613 1176 consensus_queue.cc:260] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.664960 1157 ts_tablet_manager.cc:1905] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.668040 1158 consensus_queue.cc:260] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.670995 1158 ts_tablet_manager.cc:1428] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250109 20:53:04.671294 1158 tablet_bootstrap.cc:492] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:04.675769 1158 tablet_bootstrap.cc:654] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:04.681766 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70581bd3aa674b73ab1b4908fe1c9030 consensus state has no leader
W20250109 20:53:04.682155 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6492caf3355c4685a6f0670ad1a352e9 consensus state has no leader
I20250109 20:53:04.682657 713 leader_election.cc:304] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.682910 2469 raft_consensus.cc:2798] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:04.683001 2469 raft_consensus.cc:695] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
W20250109 20:53:04.683063 713 leader_election.cc:343] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 225046d45c1c43b488babf5da8de75d7
I20250109 20:53:04.683131 713 leader_election.cc:304] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.683197 2469 consensus_queue.cc:237] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.687347 1157 ts_tablet_manager.cc:1918] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.687881 1176 ts_tablet_manager.cc:1428] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20250109 20:53:04.687908 1160 ts_tablet_manager.cc:1918] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.688200 1160 log.cc:1198] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2ca83a49523b45be9ac7556b40746a83
I20250109 20:53:04.688845 1160 ts_tablet_manager.cc:1939] T 2ca83a49523b45be9ac7556b40746a83 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:04.688951 1176 tablet_bootstrap.cc:492] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:04.690181 1176 tablet_bootstrap.cc:654] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.689446 883 tablet_service.cc:1467] Processing CreateTablet for tablet aaf2f82c48234cf7b5cfa0507e944a65 (DEFAULT_TABLE table=default.Table_f0b3bb838274407daa9df9dc44897208 [id=8bab0c83d30347e1a6f2f8fac4e52724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.690527 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaf2f82c48234cf7b5cfa0507e944a65. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:04.692349 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ae81d22bb904903a75590beca4b7ee2 consensus state has no leader
I20250109 20:53:04.689699 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb148c654514b37bd1a565284a67027 (DEFAULT_TABLE table=default.Table_10c76d34136a4a63afcc050df2123e97 [id=226b62a712284a4d909efd20323b5f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.693063 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fb148c654514b37bd1a565284a67027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.702013 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1154) took 1431 ms. Trace:
I20250109 20:53:04.702178 1160 rpcz_store.cc:276] 0109 20:53:03.270335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.270379 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:04.591814 (+1321435us) ts_tablet_manager.cc:1145] Deleting tablet 2ca83a49523b45be9ac7556b40746a83
0109 20:53:04.591830 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.668326 (+ 76496us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.687805 (+ 19479us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.701975 (+ 14170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321255}
I20250109 20:53:04.703794 890 tablet_service.cc:1467] Processing CreateTablet for tablet 49e0863249364194936804ae83645936 (DEFAULT_TABLE table=default.Table_7dbdb0f9160444ca9abff301774ac41d [id=298a30a3008046b1bb63d18f4b21e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.706676 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 49e0863249364194936804ae83645936 (DEFAULT_TABLE table=default.Table_7dbdb0f9160444ca9abff301774ac41d [id=298a30a3008046b1bb63d18f4b21e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.706882 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49e0863249364194936804ae83645936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.706864 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 2098d97e88be463c89ad3ea42e26938c (DEFAULT_TABLE table=default.Table_409e2ace4d204027a7d56be14d3f9c9c [id=cb959bbb2102410c93dfdf0bcb15248e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.707000 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49e0863249364194936804ae83645936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.707212 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2098d97e88be463c89ad3ea42e26938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.707168 1032 tablet_service.cc:1467] Processing CreateTablet for tablet df646addd3434934a62702e9ca9731c0 (DEFAULT_TABLE table=default.Table_93ab49fd9374420f80e0422073c909da [id=4451c2feae844c89bbe000829e1e69d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.706898 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d127e2502b1547189e37e4b8f0dddc1f (DEFAULT_TABLE table=default.Table_b49ca6bc758a4b8c9c39cc064d3e44c7 [id=b5f9f71304ff4e389017b6c32f1598a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.707405 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df646addd3434934a62702e9ca9731c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.707430 1017 tablet_service.cc:1467] Processing CreateTablet for tablet e9c5051b532748cdb1d218351a81513e (DEFAULT_TABLE table=default.Table_a07934849b084f54a2033b79378af24e [id=412fea3799af443a9a152595a911f471]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.707605 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d127e2502b1547189e37e4b8f0dddc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.707763 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c5051b532748cdb1d218351a81513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.710434 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb148c654514b37bd1a565284a67027 (DEFAULT_TABLE table=default.Table_10c76d34136a4a63afcc050df2123e97 [id=226b62a712284a4d909efd20323b5f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.710691 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fb148c654514b37bd1a565284a67027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.704604 888 tablet_service.cc:1467] Processing CreateTablet for tablet df646addd3434934a62702e9ca9731c0 (DEFAULT_TABLE table=default.Table_93ab49fd9374420f80e0422073c909da [id=4451c2feae844c89bbe000829e1e69d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.712121 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df646addd3434934a62702e9ca9731c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.716303 1016 tablet_service.cc:1467] Processing CreateTablet for tablet aaf2f82c48234cf7b5cfa0507e944a65 (DEFAULT_TABLE table=default.Table_f0b3bb838274407daa9df9dc44897208 [id=8bab0c83d30347e1a6f2f8fac4e52724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.704926 885 tablet_service.cc:1467] Processing CreateTablet for tablet 2098d97e88be463c89ad3ea42e26938c (DEFAULT_TABLE table=default.Table_409e2ace4d204027a7d56be14d3f9c9c [id=cb959bbb2102410c93dfdf0bcb15248e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.716958 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaf2f82c48234cf7b5cfa0507e944a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.716984 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2098d97e88be463c89ad3ea42e26938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.705817 895 tablet_service.cc:1467] Processing CreateTablet for tablet e9c5051b532748cdb1d218351a81513e (DEFAULT_TABLE table=default.Table_a07934849b084f54a2033b79378af24e [id=412fea3799af443a9a152595a911f471]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.721055 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c5051b532748cdb1d218351a81513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.722946 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_95809365f485432a9035b1a4a42d6dae" } modify_external_catalogs: true
I20250109 20:53:04.723138 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_95809365f485432a9035b1a4a42d6dae" } modify_external_catalogs: true
I20250109 20:53:04.724210 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 31af7fe649b94a44a1fdf71bdd5fd083 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:04.706135 886 tablet_service.cc:1467] Processing CreateTablet for tablet 9208a21d7a634a0b9c5301ec8195adfa (DEFAULT_TABLE table=default.Table_32c8c1c279bf47f79cabf5c3332d4076 [id=6be68a0c74354bb69ba0dc1d2228fe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.725051 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9208a21d7a634a0b9c5301ec8195adfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.725765 1166 ts_tablet_manager.cc:1918] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.725862 1166 log.cc:1198] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/fce062dc0ac144a19f8fa3eef4a099b4
I20250109 20:53:04.726262 1166 ts_tablet_manager.cc:1939] T fce062dc0ac144a19f8fa3eef4a099b4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:04.726537 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 9208a21d7a634a0b9c5301ec8195adfa (DEFAULT_TABLE table=default.Table_32c8c1c279bf47f79cabf5c3332d4076 [id=6be68a0c74354bb69ba0dc1d2228fe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.727502 1033 tablet_service.cc:1467] Processing CreateTablet for tablet f954bd7e7f4348d996d24f9157e29fec (DEFAULT_TABLE table=default.Table_18a1b7ae7954468a851982d70404d1ff [id=fca320f9620a40b3b1a5314078d7588d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.729496 1020 tablet_service.cc:1467] Processing CreateTablet for tablet a13a88398c304492b516266c8d076024 (DEFAULT_TABLE table=default.Table_776ff5756e184f24bcd08d48461ec4e5 [id=117016677e5943c49a7d0fe1ff2f6ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.730612 892 tablet_service.cc:1467] Processing CreateTablet for tablet f954bd7e7f4348d996d24f9157e29fec (DEFAULT_TABLE table=default.Table_18a1b7ae7954468a851982d70404d1ff [id=fca320f9620a40b3b1a5314078d7588d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.730741 887 tablet_service.cc:1467] Processing CreateTablet for tablet a13a88398c304492b516266c8d076024 (DEFAULT_TABLE table=default.Table_776ff5756e184f24bcd08d48461ec4e5 [id=117016677e5943c49a7d0fe1ff2f6ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.731048 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954bd7e7f4348d996d24f9157e29fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.731312 893 tablet_service.cc:1467] Processing CreateTablet for tablet 5e843079de154ee5b1b340b2d0d11c7f (DEFAULT_TABLE table=default.Table_22d4e51e64bb4059ab767ed4d99798b2 [id=e64f19e795274acd9489debdafa1f0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.731638 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e843079de154ee5b1b340b2d0d11c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.732206 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954bd7e7f4348d996d24f9157e29fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.732313 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 5e843079de154ee5b1b340b2d0d11c7f (DEFAULT_TABLE table=default.Table_22d4e51e64bb4059ab767ed4d99798b2 [id=e64f19e795274acd9489debdafa1f0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.732642 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e843079de154ee5b1b340b2d0d11c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.733198 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13a88398c304492b516266c8d076024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.734449 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13a88398c304492b516266c8d076024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.736065 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:53:04.738910 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9208a21d7a634a0b9c5301ec8195adfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.741891 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 829bec4f66c24180afd5271057a7d0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.742360 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f511041c32bb44c090802e09423715e8" } modify_external_catalogs: true
I20250109 20:53:04.742532 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f511041c32bb44c090802e09423715e8" } modify_external_catalogs: true
I20250109 20:53:04.744336 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_50f7e7cb701c4134ae4844b388bf458a" } modify_external_catalogs: true
I20250109 20:53:04.744511 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_50f7e7cb701c4134ae4844b388bf458a" } modify_external_catalogs: true
I20250109 20:53:04.745796 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1a2eba2c16714ea1951f0037668ef8f4" } modify_external_catalogs: true
I20250109 20:53:04.745934 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1a2eba2c16714ea1951f0037668ef8f4" } modify_external_catalogs: true
I20250109 20:53:04.747236 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b062364df95c4447b2e99c5af63d535d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.747730 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b062364df95c4447b2e99c5af63d535d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.748852 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1211) took 1053 ms. Trace:
I20250109 20:53:04.748939 1166 rpcz_store.cc:276] 0109 20:53:03.695806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.695879 (+ 73us) service_pool.cc:225] Handling call
0109 20:53:04.652954 (+957075us) ts_tablet_manager.cc:1145] Deleting tablet fce062dc0ac144a19f8fa3eef4a099b4
0109 20:53:04.652961 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.659336 (+ 6375us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.725693 (+ 66357us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.748835 (+ 23142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":956900}
I20250109 20:53:04.749275 1166 tablet_replica.cc:331] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:04.749243 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fe2e5d6b9d9c4efaa573ac455d4073af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.749455 1166 raft_consensus.cc:2238] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.749589 1166 raft_consensus.cc:2267] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:04.749786 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe2e5d6b9d9c4efaa573ac455d4073af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.750097 1166 ts_tablet_manager.cc:1905] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.750905 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet fce062dc0ac144a19f8fa3eef4a099b4 (table default.Table_986f5a94df1245f7930aca6efd8614a2 [id=2600eebbccdb4d098f1b48223a9edcd7]) successfully deleted
I20250109 20:53:04.752553 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 2769e5906c604bd5b14df79db765ec61 (table default.Table_f59d59762b8b4f55b3123c4df5bfd12e [id=3c9451140e48432f8591841d4dfd8751]). Expected version 5 got 0
I20250109 20:53:04.706235 882 tablet_service.cc:1467] Processing CreateTablet for tablet 2298004317b14932bf80f39568ad73da (DEFAULT_TABLE table=default.Table_83001a14d61f44ec8e047a6fe2dccd69 [id=5872d24325d343fea657e10a066c770e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.755591 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2298004317b14932bf80f39568ad73da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.757040 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 2298004317b14932bf80f39568ad73da (DEFAULT_TABLE table=default.Table_83001a14d61f44ec8e047a6fe2dccd69 [id=5872d24325d343fea657e10a066c770e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.757412 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2298004317b14932bf80f39568ad73da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.706521 883 tablet_service.cc:1467] Processing CreateTablet for tablet d127e2502b1547189e37e4b8f0dddc1f (DEFAULT_TABLE table=default.Table_b49ca6bc758a4b8c9c39cc064d3e44c7 [id=b5f9f71304ff4e389017b6c32f1598a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.761133 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d127e2502b1547189e37e4b8f0dddc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.761368 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:53:04.761626 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:04.762346 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 1c65489a602141239c99e8e8dae22ab4 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]). Expected version 4 got 0
I20250109 20:53:04.762780 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]). Expected version 1 got 0
I20250109 20:53:04.767524 888 tablet_service.cc:1467] Processing CreateTablet for tablet 85a8bfb21cb44e14bdacdf8b8208a116 (DEFAULT_TABLE table=default.Table_201a598c2da7487bbb7cd29c9a0d9ae1 [id=8ec3ba4b2d4c42d38b83dd54af59d63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.767957 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a8bfb21cb44e14bdacdf8b8208a116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.769268 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ede7e3a1d7df4082ab7af95a46349423 (DEFAULT_TABLE table=default.Table_1f012e6f7de14e748ed0497f6dfc2a00 [id=4f5b480f2af442d88e54940704721f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.769778 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede7e3a1d7df4082ab7af95a46349423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.772930 886 tablet_service.cc:1467] Processing CreateTablet for tablet ede7e3a1d7df4082ab7af95a46349423 (DEFAULT_TABLE table=default.Table_1f012e6f7de14e748ed0497f6dfc2a00 [id=4f5b480f2af442d88e54940704721f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.773370 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede7e3a1d7df4082ab7af95a46349423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.775595 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 85a8bfb21cb44e14bdacdf8b8208a116 (DEFAULT_TABLE table=default.Table_201a598c2da7487bbb7cd29c9a0d9ae1 [id=8ec3ba4b2d4c42d38b83dd54af59d63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.781010 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2ca83a49523b45be9ac7556b40746a83 (table default.Table_79f5ff4b60444c1fb9e227b39c89e4f4 [id=523cbdd3d4344d24afca6003e115d5ed]) successfully deleted
I20250109 20:53:04.781411 1160 tablet_replica.cc:331] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:04.781685 1160 raft_consensus.cc:2238] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.782130 1160 raft_consensus.cc:2267] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.782649 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 568e4a83fddd45c8b61849a369074c4d
I20250109 20:53:04.782814 1160 ts_tablet_manager.cc:1905] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.784945 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 568e4a83fddd45c8b61849a369074c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.786360 887 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2eb4c8de43470a9540162b9cf17db7 (DEFAULT_TABLE table=default.Table_502dc59bfc324b38ba72180fd2141f62 [id=8c17e96e093146beb6eeb5037ae67c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.786785 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2eb4c8de43470a9540162b9cf17db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.787037 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2eb4c8de43470a9540162b9cf17db7 (DEFAULT_TABLE table=default.Table_502dc59bfc324b38ba72180fd2141f62 [id=8c17e96e093146beb6eeb5037ae67c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.789669 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_be351430ca734ccd895e178828cb82b0" } new_table_name: "default.Table_826fa37db91b4f23ae2bc375badf74f4" modify_external_catalogs: true
I20250109 20:53:04.794236 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 070477648d3e49d0af77fbf594d8ef77 (DEFAULT_TABLE table=default.Table_38af1d150640411a90cf87f30d618cb5 [id=fd187fa50f114073ad207c12b5ebe2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.794453 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2768784b71f9461bbe4081deaaae9588" } new_table_name: "default.Table_4ea3d0314f1d4ec0a13b822cdba031ce" modify_external_catalogs: true
I20250109 20:53:04.796332 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8335c1b94e274c0d8d8e927554aa0618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.796924 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8335c1b94e274c0d8d8e927554aa0618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.800649 2470 raft_consensus.cc:491] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.800794 2470 raft_consensus.cc:513] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.801371 2470 leader_election.cc:290] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.802762 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 773ba54a01394152a54ac72239989361 (DEFAULT_TABLE table=default.Table_ff10bec6ea2f476894eda188f6b9c63d [id=0f5ec76c47684815b1408d0666976faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.805011 1166 ts_tablet_manager.cc:1918] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:04.805114 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2769e5906c604bd5b14df79db765ec61" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:04.805706 1166 log.cc:1198] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4814361a1e854451a0ee7469cc6d447b
I20250109 20:53:04.808620 2032 raft_consensus.cc:2743] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:04.810957 989 leader_election.cc:343] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.812865 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a8bfb21cb44e14bdacdf8b8208a116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.813570 2502 tablet.cc:1721] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:04.815984 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2eb4c8de43470a9540162b9cf17db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.817931 2470 raft_consensus.cc:491] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.818053 2470 raft_consensus.cc:513] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.818632 2470 leader_election.cc:290] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.819075 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070477648d3e49d0af77fbf594d8ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.830530 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65489a602141239c99e8e8dae22ab4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:04.831002 989 leader_election.cc:343] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.835320 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773ba54a01394152a54ac72239989361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.837304 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 568e4a83fddd45c8b61849a369074c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.695998 2413 raft_consensus.cc:2743] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.802325 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2769e5906c604bd5b14df79db765ec61" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.819319 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65489a602141239c99e8e8dae22ab4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:04.838053 778 raft_consensus.cc:2463] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
W20250109 20:53:04.838524 990 leader_election.cc:343] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.838624 990 leader_election.cc:304] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.838984 990 leader_election.cc:304] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:04.839334 2368 raft_consensus.cc:2743] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.839459 2368 raft_consensus.cc:2798] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:04.839500 2368 raft_consensus.cc:491] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:04.839547 2368 raft_consensus.cc:3054] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:04.840313 1166 ts_tablet_manager.cc:1939] T 4814361a1e854451a0ee7469cc6d447b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:04.841702 2368 raft_consensus.cc:513] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.842245 2368 leader_election.cc:290] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:04.843015 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65489a602141239c99e8e8dae22ab4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:04.843204 778 raft_consensus.cc:3054] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:04.843604 1176 tablet_bootstrap.cc:492] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:04.843708 1176 ts_tablet_manager.cc:1397] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.155s user 0.003s sys 0.006s
I20250109 20:53:04.844300 1176 raft_consensus.cc:357] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.844439 1176 raft_consensus.cc:383] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.844478 1176 raft_consensus.cc:738] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.844663 1176 consensus_queue.cc:260] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.845109 1176 ts_tablet_manager.cc:1428] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:53:04.845360 1176 tablet_bootstrap.cc:492] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:04.845594 778 raft_consensus.cc:2463] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:04.846125 990 leader_election.cc:304] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:04.846333 2470 raft_consensus.cc:2798] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:04.846416 2470 raft_consensus.cc:695] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:53:04.846607 2470 consensus_queue.cc:237] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.852553 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet aba42482feab4d0f85fb46b4c8b01d19
I20250109 20:53:04.857601 1165 tablet_bootstrap.cc:492] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:04.857748 1165 ts_tablet_manager.cc:1397] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.383s user 0.031s sys 0.055s
I20250109 20:53:04.858454 1165 raft_consensus.cc:357] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.858639 1165 raft_consensus.cc:383] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.858713 1165 raft_consensus.cc:738] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.859076 1165 consensus_queue.cc:260] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.859789 1165 ts_tablet_manager.cc:1428] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:04.860090 1165 tablet_bootstrap.cc:492] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:04.861588 1165 tablet_bootstrap.cc:654] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.864701 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 05979bcd1ff149a39aeb05baa093e47c (DEFAULT_TABLE table=default.Table_91c5f2662902438187bc75ecc62c5017 [id=b642b4502639480da6eab1b75d3c1d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.865111 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05979bcd1ff149a39aeb05baa093e47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.866142 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 6c16437bb00e410a9fcb774aa3d32172 (DEFAULT_TABLE table=default.Table_848a8bb910414b7faca9f9f52cbfc502 [id=30ecda085dcf451d9e7f8e07d49b8ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.866567 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c16437bb00e410a9fcb774aa3d32172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.869354 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 171c13fc9a604efc907a6d08f647d524 (DEFAULT_TABLE table=default.Table_cae85363f473456ea6dff908e3978813 [id=93096d16d4ee4ef0946565d4598e30df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.869663 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171c13fc9a604efc907a6d08f647d524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.873697 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet aba42482feab4d0f85fb46b4c8b01d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.874274 1157 log.cc:1198] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/8f420b7e623243e093d26c95276c5af9
I20250109 20:53:04.874675 1157 ts_tablet_manager.cc:1939] T 8f420b7e623243e093d26c95276c5af9 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:04.875828 1157 tablet_replica.cc:331] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:04.876101 1157 raft_consensus.cc:2238] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:04.876299 1157 raft_consensus.cc:2267] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:04.876878 1157 ts_tablet_manager.cc:1905] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.877470 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 165f6304acc849569a8e90d135a8e2d0 (table default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320]). Expected version 1 got 0
I20250109 20:53:04.877740 454 catalog_manager.cc:5526] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:04.878135 887 tablet_service.cc:1467] Processing CreateTablet for tablet 070477648d3e49d0af77fbf594d8ef77 (DEFAULT_TABLE table=default.Table_38af1d150640411a90cf87f30d618cb5 [id=fd187fa50f114073ad207c12b5ebe2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.878621 883 tablet_service.cc:1467] Processing CreateTablet for tablet 773ba54a01394152a54ac72239989361 (DEFAULT_TABLE table=default.Table_ff10bec6ea2f476894eda188f6b9c63d [id=0f5ec76c47684815b1408d0666976faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.880345 2413 raft_consensus.cc:491] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.880460 2413 raft_consensus.cc:513] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.881050 2413 leader_election.cc:290] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:04.881960 713 leader_election.cc:343] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 85a95a31468b431fb4304cf6680016da
I20250109 20:53:04.882386 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a95a31468b431fb4304cf6680016da" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:04.882879 712 leader_election.cc:343] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.882972 712 leader_election.cc:304] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.883288 2413 raft_consensus.cc:2743] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.885000 1165 tablet_bootstrap.cc:492] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:04.885216 1165 ts_tablet_manager.cc:1397] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.004s
I20250109 20:53:04.885995 1165 raft_consensus.cc:357] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.886247 1165 raft_consensus.cc:383] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.886353 1165 raft_consensus.cc:738] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.886631 1165 consensus_queue.cc:260] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.887204 1165 ts_tablet_manager.cc:1428] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:04.887465 1165 tablet_bootstrap.cc:492] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:04.889860 892 tablet_service.cc:1467] Processing CreateTablet for tablet 6c16437bb00e410a9fcb774aa3d32172 (DEFAULT_TABLE table=default.Table_848a8bb910414b7faca9f9f52cbfc502 [id=30ecda085dcf451d9e7f8e07d49b8ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.890461 885 tablet_service.cc:1467] Processing CreateTablet for tablet 05979bcd1ff149a39aeb05baa093e47c (DEFAULT_TABLE table=default.Table_91c5f2662902438187bc75ecc62c5017 [id=b642b4502639480da6eab1b75d3c1d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.890856 895 tablet_service.cc:1467] Processing CreateTablet for tablet 171c13fc9a604efc907a6d08f647d524 (DEFAULT_TABLE table=default.Table_cae85363f473456ea6dff908e3978813 [id=93096d16d4ee4ef0946565d4598e30df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.891018 1165 tablet_bootstrap.cc:654] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.896922 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65489a602141239c99e8e8dae22ab4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:04.897435 989 leader_election.cc:343] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:04.898844 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6bcf33f6ad4e51a20d878e944ecb27 (DEFAULT_TABLE table=default.Table_4b2efcf7340c405483d625749dea769b [id=fa93ca5a7f814a8aae62108580e4e499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.899257 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6bcf33f6ad4e51a20d878e944ecb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.901432 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6bcf33f6ad4e51a20d878e944ecb27 (DEFAULT_TABLE table=default.Table_4b2efcf7340c405483d625749dea769b [id=fa93ca5a7f814a8aae62108580e4e499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.902837 888 tablet_service.cc:1467] Processing CreateTablet for tablet 76030fe5b99b485ab6f35337a3bf320a (DEFAULT_TABLE table=default.Table_b66c546088d141c59f36c5a8a911fb1e [id=c7d04a7141b8489890192217c3c67cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.904405 1176 tablet_bootstrap.cc:654] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.905546 1165 tablet_bootstrap.cc:492] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:04.905748 1165 ts_tablet_manager.cc:1397] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250109 20:53:04.905993 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 76030fe5b99b485ab6f35337a3bf320a (DEFAULT_TABLE table=default.Table_b66c546088d141c59f36c5a8a911fb1e [id=c7d04a7141b8489890192217c3c67cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:04.906560 1165 raft_consensus.cc:357] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.906764 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76030fe5b99b485ab6f35337a3bf320a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.906787 1165 raft_consensus.cc:383] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.906859 1165 raft_consensus.cc:738] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.905593 1158 tablet_bootstrap.cc:492] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:04.907197 886 tablet_service.cc:1514] Processing DeleteTablet for tablet aba42482feab4d0f85fb46b4c8b01d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.907150 1165 consensus_queue.cc:260] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.907621 1158 ts_tablet_manager.cc:1397] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.236s user 0.003s sys 0.002s
I20250109 20:53:04.911767 1165 ts_tablet_manager.cc:1428] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:53:04.913182 1158 raft_consensus.cc:357] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.913348 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1210) took 1222 ms. Trace:
I20250109 20:53:04.913435 2032 raft_consensus.cc:491] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.913430 1166 rpcz_store.cc:276] 0109 20:53:03.691307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.700908 (+ 9601us) service_pool.cc:225] Handling call
0109 20:53:04.749236 (+1048328us) ts_tablet_manager.cc:1145] Deleting tablet 4814361a1e854451a0ee7469cc6d447b
0109 20:53:04.749244 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.752596 (+ 3352us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.804877 (+ 52281us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.913328 (+108451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1048128}
I20250109 20:53:04.913563 2032 raft_consensus.cc:513] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.913879 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4814361a1e854451a0ee7469cc6d447b (table default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]) successfully deleted
I20250109 20:53:04.914129 2032 leader_election.cc:290] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:04.914323 1166 tablet_replica.cc:331] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:04.914625 1166 raft_consensus.cc:2238] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:04.914875 854 leader_election.cc:343] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 73e82535780c47ca8b9cd94b013b4b42
I20250109 20:53:04.914896 1166 raft_consensus.cc:2267] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:04.915474 854 leader_election.cc:343] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 73e82535780c47ca8b9cd94b013b4b42
I20250109 20:53:04.915597 854 leader_election.cc:304] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.915918 1166 ts_tablet_manager.cc:1905] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:04.916062 2032 raft_consensus.cc:2743] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.916340 2032 raft_consensus.cc:491] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:04.916462 2032 raft_consensus.cc:513] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.917207 2032 leader_election.cc:290] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:04.917321 1165 tablet_bootstrap.cc:492] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:53:04.920428 854 leader_election.cc:343] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 9558dfe563d94482bf496e19b8a80edc
W20250109 20:53:04.920615 854 leader_election.cc:343] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 9558dfe563d94482bf496e19b8a80edc
I20250109 20:53:04.920707 854 leader_election.cc:304] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:04.921110 2032 raft_consensus.cc:2743] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:04.923482 1158 raft_consensus.cc:383] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.924731 1158 raft_consensus.cc:738] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.926084 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 31af7fe649b94a44a1fdf71bdd5fd083
I20250109 20:53:04.926625 452 catalog_manager.cc:6596] ReplaceTablet: tablet 31af7fe649b94a44a1fdf71bdd5fd083 deleted and replaced by tablet 33ffcbca06534d03ab82c02e16608d66
I20250109 20:53:04.926762 1158 consensus_queue.cc:260] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.928036 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:04.929816 1158 ts_tablet_manager.cc:1428] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250109 20:53:04.930120 1158 tablet_bootstrap.cc:492] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:04.931679 1158 tablet_bootstrap.cc:654] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:04.932771 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_54096fae81dc43028db9aba31cafb2a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.939015 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54096fae81dc43028db9aba31cafb2a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.940151 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 31af7fe649b94a44a1fdf71bdd5fd083 with delete_type TABLET_DATA_DELETED (replaced by tablet 33ffcbca06534d03ab82c02e16608d66) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:04.940578 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070477648d3e49d0af77fbf594d8ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.940834 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773ba54a01394152a54ac72239989361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.941031 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c16437bb00e410a9fcb774aa3d32172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.941326 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05979bcd1ff149a39aeb05baa093e47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.945396 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 31af7fe649b94a44a1fdf71bdd5fd083 with delete_type TABLET_DATA_DELETED (replaced by tablet 33ffcbca06534d03ab82c02e16608d66) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.950400 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76030fe5b99b485ab6f35337a3bf320a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.956840 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171c13fc9a604efc907a6d08f647d524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.959300 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6bcf33f6ad4e51a20d878e944ecb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:04.967674 1165 tablet_bootstrap.cc:654] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:04.971886 1034 consensus_meta.cc:287] T ba14bef371c444ab9d84a42137a01113 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent flushing consensus metadata: real 0.501s user 0.025s sys 0.000s
I20250109 20:53:04.976655 1176 tablet_bootstrap.cc:492] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:04.976964 1176 ts_tablet_manager.cc:1397] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.001s
I20250109 20:53:04.977669 1176 raft_consensus.cc:357] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.977907 1176 raft_consensus.cc:383] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:04.977988 1176 raft_consensus.cc:738] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:04.978233 1176 consensus_queue.cc:260] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:04.980497 1176 ts_tablet_manager.cc:1428] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:53:04.980787 1176 tablet_bootstrap.cc:492] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:04.983047 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ec1eaa6bc5a84b69840c540319265efb
I20250109 20:53:04.983486 1160 ts_tablet_manager.cc:1918] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:04.983625 1160 log.cc:1198] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/be44ffc9e371452f8d5f4174fbfa752a
I20250109 20:53:04.983779 448 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:04.983999 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5f05e17ca65243a898a7c75d3f453706
I20250109 20:53:04.983992 1160 ts_tablet_manager.cc:1939] T be44ffc9e371452f8d5f4174fbfa752a P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:04.984473 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1156) took 1690 ms. Trace:
I20250109 20:53:04.984673 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_21df3bf9edb24336a97d09e581e2548b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.985138 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21df3bf9edb24336a97d09e581e2548b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.986289 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 908781d94a054386b7d7c06c2c9fa3ca
I20250109 20:53:04.986855 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9630bc5dec99481588b309f93dccb0f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.987331 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9630bc5dec99481588b309f93dccb0f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:04.988193 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59ec9a54da2e43bfbb22637385c53a7a consensus state has no leader
I20250109 20:53:04.988512 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_578ca6a59d974fd78cc47a19a75f12da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:04.988812 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_019702682d2a460fab654ee8e4fc3463" } new_table_name: "default.Table_40b4ef2922be4c698c047787c4440d3e" modify_external_catalogs: true
W20250109 20:53:04.988960 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_578ca6a59d974fd78cc47a19a75f12da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:04.989811 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1e75afbcc4e43f4870bd4ae466e6830 consensus state has no leader
I20250109 20:53:04.990202 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_10b2d04e192c4740b1dffd0431e96880" } new_table_name: "default.Table_6888260a7ce740ba8e754d53fed1e700" modify_external_catalogs: true
I20250109 20:53:04.990689 885 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1eaa6bc5a84b69840c540319265efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.990762 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f05e17ca65243a898a7c75d3f453706 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.990689 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 908781d94a054386b7d7c06c2c9fa3ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:04.991317 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 9c3af2a00ab04b41915cffc3eb9206bc (table default.Table_14c64344192b45778141de193a1c433d [id=ffee48c2614543b981c241ea4a3dc24e]) successfully deleted
W20250109 20:53:04.991510 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d2c8a6555a8e4228a9a80b15c6880c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c8a6555a8e4228a9a80b15c6880c84 already in progress: creating tablet
I20250109 20:53:04.993927 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_31d3d34c2b5347fbb18b9b7d7e034d5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:04.994292 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31d3d34c2b5347fbb18b9b7d7e034d5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:04.997653 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_833b545ce26c4298871281f72871e989" } new_table_name: "default.Table_8f6e60c0df0549c9b1184c31b0b9d4ab" modify_external_catalogs: true
I20250109 20:53:04.984575 1160 rpcz_store.cc:276] 0109 20:53:03.293598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.293642 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:04.781337 (+1487695us) ts_tablet_manager.cc:1145] Deleting tablet be44ffc9e371452f8d5f4174fbfa752a
0109 20:53:04.781348 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.929136 (+147788us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.983382 (+ 54246us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:04.984376 (+ 994us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1487468}
I20250109 20:53:05.000206 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet be44ffc9e371452f8d5f4174fbfa752a (table default.Table_50d70e8a583c47059eafc658d73feb69 [id=ee4103ad12894523a9f260bc6ff91419]) successfully deleted
I20250109 20:53:05.000428 1176 tablet_bootstrap.cc:654] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.000766 886 tablet_service.cc:1514] Processing DeleteTablet for tablet daac99da5d10475c9de263bef27baeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.000857 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 728e9b6101e3447fb3c24b3917aa4872 with delete_type TABLET_DATA_DELETED (replaced by tablet d79437d800f44c5a8222eea7dc2cd4e3) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.000999 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1155) took 1707 ms. Trace:
I20250109 20:53:05.001073 1160 rpcz_store.cc:276] 0109 20:53:03.293495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.311563 (+ 18068us) service_pool.cc:225] Handling call
0109 20:53:04.999463 (+1687900us) ts_tablet_manager.cc:1145] Deleting tablet e80959e8c0944167b4ddebf4ebb382c5
0109 20:53:05.000518 (+ 1055us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.000954 (+ 436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1687714}
W20250109 20:53:05.001379 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e80959e8c0944167b4ddebf4ebb382c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e80959e8c0944167b4ddebf4ebb382c5 already in progress: creating tablet
I20250109 20:53:04.983794 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5bc430ff7d0847e2a9bff5aa2c007f12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.002410 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bc430ff7d0847e2a9bff5aa2c007f12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.003989 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 908781d94a054386b7d7c06c2c9fa3ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.005353 2032 raft_consensus.cc:491] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.005422 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f05e17ca65243a898a7c75d3f453706 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.005507 2032 raft_consensus.cc:513] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.005643 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1eaa6bc5a84b69840c540319265efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.006179 2032 leader_election.cc:290] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:05.007503 854 leader_election.cc:343] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: ffd3a1b1d3e44feda6dbaa8556e377fc
W20250109 20:53:05.007799 854 leader_election.cc:343] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: ffd3a1b1d3e44feda6dbaa8556e377fc
I20250109 20:53:05.007903 854 leader_election.cc:304] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:05.008221 2032 raft_consensus.cc:2743] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.010197 2470 raft_consensus.cc:491] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.010345 2470 raft_consensus.cc:513] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.011060 2470 leader_election.cc:290] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.014462 1160 tablet_replica.cc:331] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:05.014626 1160 raft_consensus.cc:2238] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:05.014585 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c865faed94c3493ea638826bdcd4253d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.014767 1160 raft_consensus.cc:2267] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:05.015003 989 leader_election.cc:343] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.015215 1160 ts_tablet_manager.cc:1905] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:05.018010 1157 ts_tablet_manager.cc:1918] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:05.018210 1157 log.cc:1198] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/b68d23fb2df743f4afa2682f6d2e3efa
I20250109 20:53:05.018549 1157 ts_tablet_manager.cc:1939] T b68d23fb2df743f4afa2682f6d2e3efa P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:05.019874 1158 tablet_bootstrap.cc:492] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.020042 1158 ts_tablet_manager.cc:1397] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
W20250109 20:53:05.028303 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09ba79b42a104782bbb2443c9ffdfb2c consensus state has no leader
I20250109 20:53:05.028442 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6d8de56aab614e7a8c51dc1d92c651db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.028601 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8ca6e16543a94e608f9e3a73747ec6d2" } modify_external_catalogs: true
I20250109 20:53:05.028798 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8ca6e16543a94e608f9e3a73747ec6d2" } modify_external_catalogs: true
W20250109 20:53:05.029278 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d8de56aab614e7a8c51dc1d92c651db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:05.030472 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a279331730c7482580650c24365c4cba consensus state has no leader
W20250109 20:53:05.034518 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8c0f5dc3c794811b55040dcc094246c consensus state has no leader
I20250109 20:53:05.043593 1176 tablet_bootstrap.cc:492] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:05.043838 1176 ts_tablet_manager.cc:1397] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250109 20:53:05.044780 1176 raft_consensus.cc:357] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.045053 1176 raft_consensus.cc:383] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.045143 1176 raft_consensus.cc:738] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.045414 1176 consensus_queue.cc:260] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.046353 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed134863bf642e79c26e5bf75b248b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.050240 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c865faed94c3493ea638826bdcd4253d" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.051108 1158 raft_consensus.cc:357] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.051260 1158 raft_consensus.cc:383] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.051604 1176 ts_tablet_manager.cc:1428] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250109 20:53:05.051697 1158 raft_consensus.cc:738] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.051921 1158 consensus_queue.cc:260] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.052460 1158 ts_tablet_manager.cc:1428] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.030s user 0.001s sys 0.000s
I20250109 20:53:05.052726 1158 tablet_bootstrap.cc:492] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:05.054296 990 leader_election.cc:343] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.054348 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_22d4e51e64bb4059ab767ed4d99798b2" } modify_external_catalogs: true
I20250109 20:53:05.054404 990 leader_election.cc:304] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.054494 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_22d4e51e64bb4059ab767ed4d99798b2" } modify_external_catalogs: true
I20250109 20:53:05.054740 2368 raft_consensus.cc:2743] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.056254 1166 ts_tablet_manager.cc:1918] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:05.056617 1166 log.cc:1198] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/9c3af2a00ab04b41915cffc3eb9206bc
I20250109 20:53:05.058568 1166 ts_tablet_manager.cc:1939] T 9c3af2a00ab04b41915cffc3eb9206bc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:05.059218 1158 tablet_bootstrap.cc:654] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.068822 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b4b31bd4072941d5aaf8c43dada3ccc7
I20250109 20:53:05.070919 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b31bd4072941d5aaf8c43dada3ccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.073383 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b31bd4072941d5aaf8c43dada3ccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.075510 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1254) took 1111 ms. Trace:
I20250109 20:53:05.075757 1157 rpcz_store.cc:276] 0109 20:53:03.963809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.963877 (+ 68us) service_pool.cc:225] Handling call
0109 20:53:04.875790 (+911913us) ts_tablet_manager.cc:1145] Deleting tablet b68d23fb2df743f4afa2682f6d2e3efa
0109 20:53:04.875796 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:04.974443 (+ 98647us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.017911 (+ 43468us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.075485 (+ 57574us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55755,"tablet-delete.queue_time_us":909373}
I20250109 20:53:05.076333 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1255) took 1102 ms. Trace:
I20250109 20:53:05.076468 1157 rpcz_store.cc:276] 0109 20:53:03.974370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.974417 (+ 47us) service_pool.cc:225] Handling call
0109 20:53:05.076215 (+1101798us) ts_tablet_manager.cc:1145] Deleting tablet e1faf5d79a9b43ffafc9c34a5937fcc0
0109 20:53:05.076226 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.076317 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101625}
I20250109 20:53:05.077083 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 801838c21f70424ba1b2d89585bda643 with delete_type TABLET_DATA_DELETED (replaced by tablet f12b13cead8942e2a4e874dde17a0802) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.077435 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d0dc4addb7fd4d23822b25af5e288118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.077745 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.077814 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 165f6304acc849569a8e90d135a8e2d0 (table default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320]). Expected version 1 got 0
I20250109 20:53:05.078047 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b087b7546a39451089666a36d029e645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.078006 447 catalog_manager.cc:5526] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
I20250109 20:53:05.078294 742 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.078341 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 1c65489a602141239c99e8e8dae22ab4 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]). Expected version 4 got 0
I20250109 20:53:05.078550 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 8539cefebb844b3098a308ca580b0e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.078976 742 tablet_service.cc:1467] Processing CreateTablet for tablet aaf2f82c48234cf7b5cfa0507e944a65 (DEFAULT_TABLE table=default.Table_f0b3bb838274407daa9df9dc44897208 [id=8bab0c83d30347e1a6f2f8fac4e52724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.079347 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaf2f82c48234cf7b5cfa0507e944a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.079576 895 tablet_service.cc:1467] Processing CreateTablet for tablet 27cb4e6e3a5944bbadb8f1d619d7361c (DEFAULT_TABLE table=default.Table_43180160d5a044548b7cf16b6c122bfa [id=c987deabe0ac45748a16183ed6408a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.079926 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cb4e6e3a5944bbadb8f1d619d7361c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.079890 883 tablet_service.cc:1467] Processing CreateTablet for tablet b6044240c05e41e29eccf27086bdc103 (DEFAULT_TABLE table=default.Table_0810c66546f749f4bde30df12a982a14 [id=ad70e6b80b1549c2a415f29a58c2bef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.080185 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6044240c05e41e29eccf27086bdc103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.084949 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5e843079de154ee5b1b340b2d0d11c7f
I20250109 20:53:05.086418 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e843079de154ee5b1b340b2d0d11c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.086562 742 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb148c654514b37bd1a565284a67027 (DEFAULT_TABLE table=default.Table_10c76d34136a4a63afcc050df2123e97 [id=226b62a712284a4d909efd20323b5f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.086848 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fb148c654514b37bd1a565284a67027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.086776 892 tablet_service.cc:1467] Processing CreateTablet for tablet e4768662a5df4adfa8d5136d46b07992 (DEFAULT_TABLE table=default.Table_c91107370a3c405c8532e0931e7440c2 [id=9cb902752f004b60b48928b50753e528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.086874 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 165f6304acc849569a8e90d135a8e2d0 (table default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320]). Expected version 1 got 0
I20250109 20:53:05.087039 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4768662a5df4adfa8d5136d46b07992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.089068 754 tablet_service.cc:1467] Processing CreateTablet for tablet 2298004317b14932bf80f39568ad73da (DEFAULT_TABLE table=default.Table_83001a14d61f44ec8e047a6fe2dccd69 [id=5872d24325d343fea657e10a066c770e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.089537 755 tablet_service.cc:1467] Processing CreateTablet for tablet d127e2502b1547189e37e4b8f0dddc1f (DEFAULT_TABLE table=default.Table_b49ca6bc758a4b8c9c39cc064d3e44c7 [id=b5f9f71304ff4e389017b6c32f1598a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.089968 751 tablet_service.cc:1467] Processing CreateTablet for tablet 49e0863249364194936804ae83645936 (DEFAULT_TABLE table=default.Table_7dbdb0f9160444ca9abff301774ac41d [id=298a30a3008046b1bb63d18f4b21e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.090345 750 tablet_service.cc:1467] Processing CreateTablet for tablet df646addd3434934a62702e9ca9731c0 (DEFAULT_TABLE table=default.Table_93ab49fd9374420f80e0422073c909da [id=4451c2feae844c89bbe000829e1e69d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.090695 747 tablet_service.cc:1467] Processing CreateTablet for tablet 2098d97e88be463c89ad3ea42e26938c (DEFAULT_TABLE table=default.Table_409e2ace4d204027a7d56be14d3f9c9c [id=cb959bbb2102410c93dfdf0bcb15248e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.091027 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2298004317b14932bf80f39568ad73da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.091104 739 tablet_service.cc:1467] Processing CreateTablet for tablet e9c5051b532748cdb1d218351a81513e (DEFAULT_TABLE table=default.Table_a07934849b084f54a2033b79378af24e [id=412fea3799af443a9a152595a911f471]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.091490 756 tablet_service.cc:1467] Processing CreateTablet for tablet 9208a21d7a634a0b9c5301ec8195adfa (DEFAULT_TABLE table=default.Table_32c8c1c279bf47f79cabf5c3332d4076 [id=6be68a0c74354bb69ba0dc1d2228fe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.091893 740 tablet_service.cc:1467] Processing CreateTablet for tablet f954bd7e7f4348d996d24f9157e29fec (DEFAULT_TABLE table=default.Table_18a1b7ae7954468a851982d70404d1ff [id=fca320f9620a40b3b1a5314078d7588d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.092743 753 tablet_service.cc:1467] Processing CreateTablet for tablet 85a8bfb21cb44e14bdacdf8b8208a116 (DEFAULT_TABLE table=default.Table_201a598c2da7487bbb7cd29c9a0d9ae1 [id=8ec3ba4b2d4c42d38b83dd54af59d63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.092908 745 tablet_service.cc:1467] Processing CreateTablet for tablet a13a88398c304492b516266c8d076024 (DEFAULT_TABLE table=default.Table_776ff5756e184f24bcd08d48461ec4e5 [id=117016677e5943c49a7d0fe1ff2f6ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.092955 1020 tablet_service.cc:1467] Processing CreateTablet for tablet b6044240c05e41e29eccf27086bdc103 (DEFAULT_TABLE table=default.Table_0810c66546f749f4bde30df12a982a14 [id=ad70e6b80b1549c2a415f29a58c2bef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.093024 757 tablet_service.cc:1467] Processing CreateTablet for tablet ede7e3a1d7df4082ab7af95a46349423 (DEFAULT_TABLE table=default.Table_1f012e6f7de14e748ed0497f6dfc2a00 [id=4f5b480f2af442d88e54940704721f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.093221 746 tablet_service.cc:1467] Processing CreateTablet for tablet 5e843079de154ee5b1b340b2d0d11c7f (DEFAULT_TABLE table=default.Table_22d4e51e64bb4059ab767ed4d99798b2 [id=e64f19e795274acd9489debdafa1f0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.093868 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 6964d30ce23c408f8f6caa7dceb58462 (DEFAULT_TABLE table=default.Table_bc513c9e60954461ad69deebb17af802 [id=21bf9ae041de46ed83ba0ff3ddaf6ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.093964 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e843079de154ee5b1b340b2d0d11c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.094203 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 46c10c113a114ff1b48256844261191a (DEFAULT_TABLE table=default.Table_75c0dbbb51764601811f6bff5ca923c4 [id=79a2c1aa694b45e8b8cfbe575c4fcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.094300 1017 tablet_service.cc:1467] Processing CreateTablet for tablet e4768662a5df4adfa8d5136d46b07992 (DEFAULT_TABLE table=default.Table_c91107370a3c405c8532e0931e7440c2 [id=9cb902752f004b60b48928b50753e528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.095168 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 27cb4e6e3a5944bbadb8f1d619d7361c (DEFAULT_TABLE table=default.Table_43180160d5a044548b7cf16b6c122bfa [id=c987deabe0ac45748a16183ed6408a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.096490 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2098d97e88be463c89ad3ea42e26938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.096784 1032 tablet_service.cc:1467] Processing CreateTablet for tablet f12b13cead8942e2a4e874dde17a0802 (DEFAULT_TABLE table=default.Table_261f03425c0e4e77873973fa3746a8c3 [id=ae99a336de5a4336bfaa8e4d2f8994e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.097101 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 506bead02bb4424d8b5b7b30abf190f1 (DEFAULT_TABLE table=default.Table_1f2b650f7699459eadc01979a7bc8b90 [id=2d986b1e5528483c9675133605225d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.097384 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 932e769ac14b497aa9187338cbc8b986 (DEFAULT_TABLE table=default.Table_75ec2484861c48b2afc25351a61ba546 [id=e9c0cdde36994e8194e5a2f0f63e1381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.097640 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6880a3e914414c77bf7682da4057df4c (DEFAULT_TABLE table=default.Table_1f0817571af249be925827a9c80c6ffd [id=beed9ffd0f694e0db99c29196c360f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.097900 1033 tablet_service.cc:1467] Processing CreateTablet for tablet e319004d8ff94576bc1c7be3c71baf9b (DEFAULT_TABLE table=default.Table_a2f187de26b34ff2a1d1be7289628fcc [id=0599ff264e4444399e082ae50ea05593]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.098788 887 tablet_service.cc:1467] Processing CreateTablet for tablet f12b13cead8942e2a4e874dde17a0802 (DEFAULT_TABLE table=default.Table_261f03425c0e4e77873973fa3746a8c3 [id=ae99a336de5a4336bfaa8e4d2f8994e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.099143 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f12b13cead8942e2a4e874dde17a0802. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.099378 885 tablet_service.cc:1467] Processing CreateTablet for tablet 506bead02bb4424d8b5b7b30abf190f1 (DEFAULT_TABLE table=default.Table_1f2b650f7699459eadc01979a7bc8b90 [id=2d986b1e5528483c9675133605225d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.099653 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506bead02bb4424d8b5b7b30abf190f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.100929 893 tablet_service.cc:1467] Processing CreateTablet for tablet 932e769ac14b497aa9187338cbc8b986 (DEFAULT_TABLE table=default.Table_75ec2484861c48b2afc25351a61ba546 [id=e9c0cdde36994e8194e5a2f0f63e1381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.101191 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932e769ac14b497aa9187338cbc8b986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.101481 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet cc54e97e91c54afea098624bd85fcc6e (table default.Table_19aea38f8d4547399b10df16cdb1e54e [id=28723b313bd34951826646276c761571]). Expected version 2 got 0
I20250109 20:53:05.103919 882 tablet_service.cc:1467] Processing CreateTablet for tablet 46c10c113a114ff1b48256844261191a (DEFAULT_TABLE table=default.Table_75c0dbbb51764601811f6bff5ca923c4 [id=79a2c1aa694b45e8b8cfbe575c4fcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.104185 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c10c113a114ff1b48256844261191a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.104974 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6880a3e914414c77bf7682da4057df4c (DEFAULT_TABLE table=default.Table_1f0817571af249be925827a9c80c6ffd [id=beed9ffd0f694e0db99c29196c360f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.105234 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6880a3e914414c77bf7682da4057df4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.109218 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c5051b532748cdb1d218351a81513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.109383 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df646addd3434934a62702e9ca9731c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.112079 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9208a21d7a634a0b9c5301ec8195adfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.112156 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet deedac68322e460b8d62adbe849de3f2 with delete_type TABLET_DATA_DELETED (replaced by tablet be05de223c3946c6b8a4899cf696fe26) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.112231 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede7e3a1d7df4082ab7af95a46349423. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.113150 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 84a7f68d163a41cea6dc4f40cf89f3eb (DEFAULT_TABLE table=default.Table_dc2216d87ffe4dc288a86a9b772341d6 [id=89c2b535c2954f518e3a18451bae432a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.114780 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f954bd7e7f4348d996d24f9157e29fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.114926 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49e0863249364194936804ae83645936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.116060 888 tablet_service.cc:1467] Processing CreateTablet for tablet 84a7f68d163a41cea6dc4f40cf89f3eb (DEFAULT_TABLE table=default.Table_dc2216d87ffe4dc288a86a9b772341d6 [id=89c2b535c2954f518e3a18451bae432a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.116411 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a7f68d163a41cea6dc4f40cf89f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.116576 2470 raft_consensus.cc:491] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.116695 2470 raft_consensus.cc:513] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.116994 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a8bfb21cb44e14bdacdf8b8208a116. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.117270 2470 leader_election.cc:290] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.117655 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829bec4f66c24180afd5271057a7d0e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:05.118108 989 leader_election.cc:343] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.119498 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d127e2502b1547189e37e4b8f0dddc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.120285 886 tablet_service.cc:1467] Processing CreateTablet for tablet e319004d8ff94576bc1c7be3c71baf9b (DEFAULT_TABLE table=default.Table_a2f187de26b34ff2a1d1be7289628fcc [id=0599ff264e4444399e082ae50ea05593]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.120545 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e319004d8ff94576bc1c7be3c71baf9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.121366 895 tablet_service.cc:1467] Processing CreateTablet for tablet 6964d30ce23c408f8f6caa7dceb58462 (DEFAULT_TABLE table=default.Table_bc513c9e60954461ad69deebb17af802 [id=21bf9ae041de46ed83ba0ff3ddaf6ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.121620 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6964d30ce23c408f8f6caa7dceb58462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.122058 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e843079de154ee5b1b340b2d0d11c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.125083 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13a88398c304492b516266c8d076024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.126330 1158 tablet_bootstrap.cc:492] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.126439 1158 ts_tablet_manager.cc:1397] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20250109 20:53:05.127157 1158 raft_consensus.cc:357] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.127316 1158 raft_consensus.cc:383] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.127355 1158 raft_consensus.cc:738] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.127477 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829bec4f66c24180afd5271057a7d0e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.127524 1158 consensus_queue.cc:260] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:05.127960 990 leader_election.cc:343] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.128077 990 leader_election.cc:304] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.128520 2470 raft_consensus.cc:2743] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.130467 1158 ts_tablet_manager.cc:1428] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:53:05.130800 1158 tablet_bootstrap.cc:492] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:05.131964 1158 tablet_bootstrap.cc:654] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.143795 447 master_service.cc:946] ReplaceTablet: received request to replace tablet b54b66ca10114330b3485f67eb076064 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:05.149466 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b54b66ca10114330b3485f67eb076064
I20250109 20:53:05.150563 447 catalog_manager.cc:6596] ReplaceTablet: tablet b54b66ca10114330b3485f67eb076064 deleted and replaced by tablet 9719bbee9dbc4934992b47bf7b9bd249
I20250109 20:53:05.150619 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b54b66ca10114330b3485f67eb076064 with delete_type TABLET_DATA_DELETED (replaced by tablet 9719bbee9dbc4934992b47bf7b9bd249) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.151252 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1256) took 1169 ms. Trace:
I20250109 20:53:05.151373 1157 rpcz_store.cc:276] 0109 20:53:03.981642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.981687 (+ 45us) service_pool.cc:225] Handling call
0109 20:53:05.151123 (+1169436us) ts_tablet_manager.cc:1145] Deleting tablet 8f2ba81d736745fb9d56fd9a7c229153
0109 20:53:05.151130 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.151236 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1169290}
I20250109 20:53:05.151674 1157 tablet_replica.cc:331] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:05.151822 895 tablet_service.cc:1514] Processing DeleteTablet for tablet b54b66ca10114330b3485f67eb076064 with delete_type TABLET_DATA_DELETED (replaced by tablet 9719bbee9dbc4934992b47bf7b9bd249) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.151887 1157 raft_consensus.cc:2238] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:05.152015 1157 raft_consensus.cc:2267] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:05.152549 1157 ts_tablet_manager.cc:1905] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:05.158319 1176 tablet_bootstrap.cc:492] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:05.158497 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6964d30ce23c408f8f6caa7dceb58462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.159664 1176 tablet_bootstrap.cc:654] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.163714 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c10c113a114ff1b48256844261191a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.163998 2032 raft_consensus.cc:491] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.164121 2032 raft_consensus.cc:513] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.164685 2032 leader_election.cc:290] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.166677 1158 tablet_bootstrap.cc:492] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.166807 1158 ts_tablet_manager.cc:1397] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250109 20:53:05.167649 1158 raft_consensus.cc:357] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.167846 1158 raft_consensus.cc:383] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.167910 1158 raft_consensus.cc:738] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.168162 1158 consensus_queue.cc:260] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:05.172885 854 leader_election.cc:343] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 8ed134863bf642e79c26e5bf75b248b0
I20250109 20:53:05.175076 1158 ts_tablet_manager.cc:1428] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250109 20:53:05.175347 1158 tablet_bootstrap.cc:492] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:05.175520 854 leader_election.cc:343] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8ed134863bf642e79c26e5bf75b248b0
I20250109 20:53:05.175621 854 leader_election.cc:304] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:05.175993 2032 raft_consensus.cc:2743] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.177086 1158 tablet_bootstrap.cc:654] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.179069 1165 tablet_bootstrap.cc:492] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:05.179324 1165 ts_tablet_manager.cc:1397] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.262s user 0.012s sys 0.029s
I20250109 20:53:05.182129 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet cc54e97e91c54afea098624bd85fcc6e (table default.Table_19aea38f8d4547399b10df16cdb1e54e [id=28723b313bd34951826646276c761571]). Expected version 2 got 0
I20250109 20:53:05.182042 1165 raft_consensus.cc:357] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.183045 2470 raft_consensus.cc:491] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.183173 2470 raft_consensus.cc:513] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.182974 1165 raft_consensus.cc:383] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.183362 1165 raft_consensus.cc:738] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.183732 2470 leader_election.cc:290] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.184365 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc235614171f4a61b75492ddeb3f0fe6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.184448 1165 consensus_queue.cc:260] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.185750 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc235614171f4a61b75492ddeb3f0fe6" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.186055 1176 tablet_bootstrap.cc:492] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:05.186242 1176 ts_tablet_manager.cc:1397] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
W20250109 20:53:05.186578 989 leader_election.cc:343] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:05.186836 990 leader_election.cc:343] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.186926 990 leader_election.cc:304] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.186961 1176 raft_consensus.cc:357] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.187119 1176 raft_consensus.cc:383] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.187165 1176 raft_consensus.cc:738] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.187366 2470 raft_consensus.cc:2743] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.187322 1176 consensus_queue.cc:260] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.188017 1176 ts_tablet_manager.cc:1428] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:05.194834 895 tablet_service.cc:1514] Processing DeleteTablet for tablet ca816507d0c047668a51f6e7ebce18cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce0a941f0c14501a947a5c7e2a844e8) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.196799 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_503a302a104946dba7674a2aba9dba11" } modify_external_catalogs: true
I20250109 20:53:05.196978 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_503a302a104946dba7674a2aba9dba11" } modify_external_catalogs: true
I20250109 20:53:05.198202 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f493c72deafc46099ed5a76527351d35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.198660 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f493c72deafc46099ed5a76527351d35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.199944 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_18a1b7ae7954468a851982d70404d1ff" } modify_external_catalogs: true
I20250109 20:53:05.200081 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_18a1b7ae7954468a851982d70404d1ff" } modify_external_catalogs: true
I20250109 20:53:05.201272 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d6ff26e18b7c4cb2a89fcb0a7338382b" } new_table_name: "default.Table_b2c8d2b0a7794d608e3a0f1aab3c0e77" modify_external_catalogs: true
I20250109 20:53:05.202347 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_48be3078038a4e47a7ece82d379d0c41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.202754 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48be3078038a4e47a7ece82d379d0c41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.202879 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_63f5bc8b110a4e77b65f5074015ab1cd" } modify_external_catalogs: true
I20250109 20:53:05.203004 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_63f5bc8b110a4e77b65f5074015ab1cd" } modify_external_catalogs: true
I20250109 20:53:05.204048 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2fb311daf94d4ba582536280f88fd7b0" } new_table_name: "default.Table_0fb4f70801f04035bbef8a3b62b224da" modify_external_catalogs: true
I20250109 20:53:05.204160 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4d2d7ac2ad254b55a416b717dfd03e11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.204838 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d2d7ac2ad254b55a416b717dfd03e11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.207310 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cb4e6e3a5944bbadb8f1d619d7361c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.207063 1165 ts_tablet_manager.cc:1428] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20250109 20:53:05.211968 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b0d3e7f598ca4696b733abd94238f85f
I20250109 20:53:05.212505 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f12b13cead8942e2a4e874dde17a0802. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.216961 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506bead02bb4424d8b5b7b30abf190f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.217938 1165 tablet_bootstrap.cc:492] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:05.221943 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 6461f153a5994412ba05c3b687963fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.224124 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932e769ac14b497aa9187338cbc8b986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.224733 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6880a3e914414c77bf7682da4057df4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.228919 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e319004d8ff94576bc1c7be3c71baf9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.229113 895 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3e7f598ca4696b733abd94238f85f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.231741 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f954bd7e7f4348d996d24f9157e29fec
I20250109 20:53:05.232308 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a7f68d163a41cea6dc4f40cf89f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.233036 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f954bd7e7f4348d996d24f9157e29fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.235580 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1208) took 1562 ms. Trace:
I20250109 20:53:05.235663 1166 rpcz_store.cc:276] 0109 20:53:03.673493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.701140 (+ 27647us) service_pool.cc:225] Handling call
0109 20:53:04.914276 (+1213136us) ts_tablet_manager.cc:1145] Deleting tablet 9c3af2a00ab04b41915cffc3eb9206bc
0109 20:53:04.914287 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.005857 (+ 91570us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.055599 (+ 49742us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.235563 (+179964us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212864}
I20250109 20:53:05.235953 1166 tablet_replica.cc:331] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:05.236115 1166 raft_consensus.cc:2238] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:05.236238 1166 raft_consensus.cc:2267] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:05.236691 1166 ts_tablet_manager.cc:1905] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:05.236800 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3155ce67d2a64a1c8ebd1a308b8e33c7 consensus state has no leader
I20250109 20:53:05.238603 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4768662a5df4adfa8d5136d46b07992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.239259 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cea365f9151b49bd829baf4014653bd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.239451 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_22859e47f7314012b11a0b937538dd3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.239863 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cea365f9151b49bd829baf4014653bd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:05.239899 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22859e47f7314012b11a0b937538dd3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.240437 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_045b529f170543478afcac3c58958866" } new_table_name: "default.Table_d23131ea9b10461a9b98e2c1c1e743fe" modify_external_catalogs: true
I20250109 20:53:05.241300 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a5ce359456d84af3baf8a6fccc5a82cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.241647 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 9c3af2a00ab04b41915cffc3eb9206bc (table default.Table_14c64344192b45778141de193a1c433d [id=ffee48c2614543b981c241ea4a3dc24e]) successfully deleted
W20250109 20:53:05.241797 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5ce359456d84af3baf8a6fccc5a82cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.242539 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d476f01f037441babd8bd5d6c4e53bb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.242905 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d476f01f037441babd8bd5d6c4e53bb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.243712 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_29a0e6a6805a4ffa952d1bd55e846a54" } new_table_name: "default.Table_168790ba2811484e977ede30bf28842b" modify_external_catalogs: true
I20250109 20:53:05.244750 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f9145358b66d489788ec5cb9d4cd68e5
I20250109 20:53:05.245782 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f9145358b66d489788ec5cb9d4cd68e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.245952 1176 tablet_bootstrap.cc:492] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:05.246585 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3e7f598ca4696b733abd94238f85f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.246775 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f9145358b66d489788ec5cb9d4cd68e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.247481 1176 tablet_bootstrap.cc:654] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.250275 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6044240c05e41e29eccf27086bdc103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.254571 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1a37a751353247b3a261c2adc68a6034"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.255092 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a37a751353247b3a261c2adc68a6034 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.256436 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d1a93305dc21489bb4cc1bb5f36ed408"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.256920 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1a93305dc21489bb4cc1bb5f36ed408 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.258622 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_beeef07faf3a416aa68ffed7aab3a7b7" } new_table_name: "default.Table_f2522f4235e0464f93ab8fa57d5c7920" modify_external_catalogs: true
I20250109 20:53:05.262758 1158 tablet_bootstrap.cc:492] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.262907 1158 ts_tablet_manager.cc:1397] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.008s
I20250109 20:53:05.263414 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet f954bd7e7f4348d996d24f9157e29fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.263612 1158 raft_consensus.cc:357] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.263778 1158 raft_consensus.cc:383] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.263830 1158 raft_consensus.cc:738] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.264024 1165 tablet_bootstrap.cc:654] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.264017 1158 consensus_queue.cc:260] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.264461 1158 ts_tablet_manager.cc:1428] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:05.264762 1158 tablet_bootstrap.cc:492] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:05.266234 1158 tablet_bootstrap.cc:654] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.269356 2512 tablet.cc:1721] T 165f6304acc849569a8e90d135a8e2d0 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:05.270303 2513 tablet.cc:1721] T 165f6304acc849569a8e90d135a8e2d0 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:05.270597 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_efdc83e87d8b455696e8c0aaffb7ed87" } modify_external_catalogs: true
I20250109 20:53:05.270736 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_efdc83e87d8b455696e8c0aaffb7ed87" } modify_external_catalogs: true
I20250109 20:53:05.276646 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_31f7e4437ce94a0eb4369af44e5c466f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.276860 1158 tablet_bootstrap.cc:492] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.277005 1158 ts_tablet_manager.cc:1397] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20250109 20:53:05.277671 1158 raft_consensus.cc:357] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.277899 1158 raft_consensus.cc:383] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.278000 1158 raft_consensus.cc:738] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:05.278265 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31f7e4437ce94a0eb4369af44e5c466f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.278241 1158 consensus_queue.cc:260] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.278908 1158 ts_tablet_manager.cc:1428] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:05.279170 1158 tablet_bootstrap.cc:492] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:05.280339 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 568e4a83fddd45c8b61849a369074c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.280583 1158 tablet_bootstrap.cc:654] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.280860 747 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2eb4c8de43470a9540162b9cf17db7 (DEFAULT_TABLE table=default.Table_502dc59bfc324b38ba72180fd2141f62 [id=8c17e96e093146beb6eeb5037ae67c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.281533 757 tablet_service.cc:1514] Processing DeleteTablet for tablet aba42482feab4d0f85fb46b4c8b01d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.281698 756 tablet_service.cc:1467] Processing CreateTablet for tablet 070477648d3e49d0af77fbf594d8ef77 (DEFAULT_TABLE table=default.Table_38af1d150640411a90cf87f30d618cb5 [id=fd187fa50f114073ad207c12b5ebe2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.282033 757 tablet_service.cc:1467] Processing CreateTablet for tablet 76030fe5b99b485ab6f35337a3bf320a (DEFAULT_TABLE table=default.Table_b66c546088d141c59f36c5a8a911fb1e [id=c7d04a7141b8489890192217c3c67cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.282105 740 tablet_service.cc:1467] Processing CreateTablet for tablet 773ba54a01394152a54ac72239989361 (DEFAULT_TABLE table=default.Table_ff10bec6ea2f476894eda188f6b9c63d [id=0f5ec76c47684815b1408d0666976faa]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:05.282352 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0634d1eb0064e9d85b83ea3506cc597 consensus state has no leader
I20250109 20:53:05.282541 745 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6bcf33f6ad4e51a20d878e944ecb27 (DEFAULT_TABLE table=default.Table_4b2efcf7340c405483d625749dea769b [id=fa93ca5a7f814a8aae62108580e4e499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.282832 753 tablet_service.cc:1467] Processing CreateTablet for tablet 6c16437bb00e410a9fcb774aa3d32172 (DEFAULT_TABLE table=default.Table_848a8bb910414b7faca9f9f52cbfc502 [id=30ecda085dcf451d9e7f8e07d49b8ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.283021 739 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1eaa6bc5a84b69840c540319265efb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.283164 750 tablet_service.cc:1467] Processing CreateTablet for tablet 05979bcd1ff149a39aeb05baa093e47c (DEFAULT_TABLE table=default.Table_91c5f2662902438187bc75ecc62c5017 [id=b642b4502639480da6eab1b75d3c1d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.283316 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f05e17ca65243a898a7c75d3f453706 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.283459 746 tablet_service.cc:1467] Processing CreateTablet for tablet 171c13fc9a604efc907a6d08f647d524 (DEFAULT_TABLE table=default.Table_cae85363f473456ea6dff908e3978813 [id=93096d16d4ee4ef0946565d4598e30df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.283576 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 908781d94a054386b7d7c06c2c9fa3ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:04 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.283849 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a95a31468b431fb4304cf6680016da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.284011 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 31af7fe649b94a44a1fdf71bdd5fd083 with delete_type TABLET_DATA_DELETED (replaced by tablet 33ffcbca06534d03ab82c02e16608d66) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.288364 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_07a5579463ed4880b5f06a89dfb2c469" } modify_external_catalogs: true
I20250109 20:53:05.288532 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_07a5579463ed4880b5f06a89dfb2c469" } modify_external_catalogs: true
I20250109 20:53:05.290634 2223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Bootstrap replaying log segment 1/1 (133.5K/160.5K this segment, stats: ops{read=719 overwritten=0 applied=699 ignored=0} inserts{seen=659 ignored=0} mutations{seen=798 ignored=0} orphaned_commits=0)
I20250109 20:53:05.291198 739 tablet_service.cc:1467] Processing CreateTablet for tablet 84a7f68d163a41cea6dc4f40cf89f3eb (DEFAULT_TABLE table=default.Table_dc2216d87ffe4dc288a86a9b772341d6 [id=89c2b535c2954f518e3a18451bae432a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.291533 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b31bd4072941d5aaf8c43dada3ccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.292160 751 tablet_service.cc:1467] Processing CreateTablet for tablet 27cb4e6e3a5944bbadb8f1d619d7361c (DEFAULT_TABLE table=default.Table_43180160d5a044548b7cf16b6c122bfa [id=c987deabe0ac45748a16183ed6408a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.296574 1166 ts_tablet_manager.cc:1918] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:05.292241 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e843079de154ee5b1b340b2d0d11c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.296694 1166 log.cc:1198] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/de8ea5ea7fa44866b6d7160466f5a896
I20250109 20:53:05.292531 755 tablet_service.cc:1467] Processing CreateTablet for tablet 6964d30ce23c408f8f6caa7dceb58462 (DEFAULT_TABLE table=default.Table_bc513c9e60954461ad69deebb17af802 [id=21bf9ae041de46ed83ba0ff3ddaf6ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.296975 1166 ts_tablet_manager.cc:1939] T de8ea5ea7fa44866b6d7160466f5a896 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:05.298164 1158 tablet_bootstrap.cc:492] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.298326 1158 ts_tablet_manager.cc:1397] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20250109 20:53:05.298609 742 tablet_service.cc:1467] Processing CreateTablet for tablet b6044240c05e41e29eccf27086bdc103 (DEFAULT_TABLE table=default.Table_0810c66546f749f4bde30df12a982a14 [id=ad70e6b80b1549c2a415f29a58c2bef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.299026 1158 raft_consensus.cc:357] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.299206 1158 raft_consensus.cc:383] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.299278 1158 raft_consensus.cc:738] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.299479 1158 consensus_queue.cc:260] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.300063 1158 ts_tablet_manager.cc:1428] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:05.300300 1158 tablet_bootstrap.cc:492] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:05.301767 1158 tablet_bootstrap.cc:654] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.307098 754 tablet_service.cc:1467] Processing CreateTablet for tablet e4768662a5df4adfa8d5136d46b07992 (DEFAULT_TABLE table=default.Table_c91107370a3c405c8532e0931e7440c2 [id=9cb902752f004b60b48928b50753e528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.307695 752 tablet_service.cc:1467] Processing CreateTablet for tablet 46c10c113a114ff1b48256844261191a (DEFAULT_TABLE table=default.Table_75c0dbbb51764601811f6bff5ca923c4 [id=79a2c1aa694b45e8b8cfbe575c4fcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.307829 741 tablet_service.cc:1467] Processing CreateTablet for tablet f12b13cead8942e2a4e874dde17a0802 (DEFAULT_TABLE table=default.Table_261f03425c0e4e77873973fa3746a8c3 [id=ae99a336de5a4336bfaa8e4d2f8994e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.308138 758 tablet_service.cc:1467] Processing CreateTablet for tablet e319004d8ff94576bc1c7be3c71baf9b (DEFAULT_TABLE table=default.Table_a2f187de26b34ff2a1d1be7289628fcc [id=0599ff264e4444399e082ae50ea05593]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.308310 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 82209a6b39bb48e791ff63a8167c9765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.308781 749 tablet_service.cc:1467] Processing CreateTablet for tablet 506bead02bb4424d8b5b7b30abf190f1 (DEFAULT_TABLE table=default.Table_1f2b650f7699459eadc01979a7bc8b90 [id=2d986b1e5528483c9675133605225d37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.309428 748 tablet_service.cc:1467] Processing CreateTablet for tablet 932e769ac14b497aa9187338cbc8b986 (DEFAULT_TABLE table=default.Table_75ec2484861c48b2afc25351a61ba546 [id=e9c0cdde36994e8194e5a2f0f63e1381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.309499 744 tablet_service.cc:1467] Processing CreateTablet for tablet 6880a3e914414c77bf7682da4057df4c (DEFAULT_TABLE table=default.Table_1f0817571af249be925827a9c80c6ffd [id=beed9ffd0f694e0db99c29196c360f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.309875 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cede2a1882946a181b18d66f12c5114 with delete_type TABLET_DATA_DELETED (replaced by tablet 1343e8e822684fea8a13d8df9ec33779) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.310237 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f953f43877c4a738ffd8255afbf8275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.310588 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b54b66ca10114330b3485f67eb076064 with delete_type TABLET_DATA_DELETED (replaced by tablet 9719bbee9dbc4934992b47bf7b9bd249) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.311012 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 724d15e05a864852a118badae13a5d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.311436 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3e7f598ca4696b733abd94238f85f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.311789 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f954bd7e7f4348d996d24f9157e29fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.314913 1165 tablet_bootstrap.cc:492] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:05.316421 1165 ts_tablet_manager.cc:1397] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.001s
I20250109 20:53:05.319424 1160 ts_tablet_manager.cc:1918] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:05.320040 1160 log.cc:1198] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/06e615ce8e124cfe9432778b6b58e117
I20250109 20:53:05.320590 1160 ts_tablet_manager.cc:1939] T 06e615ce8e124cfe9432778b6b58e117 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:05.320984 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2eb4c8de43470a9540162b9cf17db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.321513 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070477648d3e49d0af77fbf594d8ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.321954 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76030fe5b99b485ab6f35337a3bf320a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.322345 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773ba54a01394152a54ac72239989361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.322757 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05979bcd1ff149a39aeb05baa093e47c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:05.323064 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d0059863d73460a9e7d12c46a7616f3 consensus state has no leader
I20250109 20:53:05.323160 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c16437bb00e410a9fcb774aa3d32172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.323518 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171c13fc9a604efc907a6d08f647d524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.323738 1158 tablet_bootstrap.cc:492] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.323843 1158 ts_tablet_manager.cc:1397] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20250109 20:53:05.323858 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6bcf33f6ad4e51a20d878e944ecb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.324255 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cb4e6e3a5944bbadb8f1d619d7361c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.324486 1158 raft_consensus.cc:357] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.324656 1158 raft_consensus.cc:383] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.324731 1158 raft_consensus.cc:738] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.324931 1158 consensus_queue.cc:260] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.324659 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6964d30ce23c408f8f6caa7dceb58462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.325631 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6880a3e914414c77bf7682da4057df4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.325644 1158 ts_tablet_manager.cc:1428] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:05.325934 1158 tablet_bootstrap.cc:492] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:05.326138 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932e769ac14b497aa9187338cbc8b986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.326499 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c10c113a114ff1b48256844261191a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.326988 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f12b13cead8942e2a4e874dde17a0802. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.327533 1158 tablet_bootstrap.cc:654] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.327477 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4768662a5df4adfa8d5136d46b07992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.328565 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e319004d8ff94576bc1c7be3c71baf9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.328881 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506bead02bb4424d8b5b7b30abf190f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.329457 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a7f68d163a41cea6dc4f40cf89f3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.352082 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:05.358498 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0e5377172f478cbefcc978ed2a9e0d (DEFAULT_TABLE table=default.Table_31d3d34c2b5347fbb18b9b7d7e034d5b [id=8bdc8ffeaf5e4930b8782cc4d9d89ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.359237 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 3f547fb556564da59df6b3743da0288e (DEFAULT_TABLE table=default.Table_54096fae81dc43028db9aba31cafb2a1 [id=863af0319a3a4e10a512bd0c2a086ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.359731 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 33ffcbca06534d03ab82c02e16608d66 (DEFAULT_TABLE table=default.Table_7768e94e2bff455eb490b7cd4e6f1fbb [id=e4d720450c6141fb82420e40b69c13ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.360442 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1212) took 1653 ms. Trace:
I20250109 20:53:05.360524 1166 rpcz_store.cc:276] 0109 20:53:03.706803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.706859 (+ 56us) service_pool.cc:225] Handling call
0109 20:53:05.235916 (+1529057us) ts_tablet_manager.cc:1145] Deleting tablet de8ea5ea7fa44866b6d7160466f5a896
0109 20:53:05.235924 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.292503 (+ 56579us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.295899 (+ 3396us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.360426 (+ 64527us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528858}
I20250109 20:53:05.360971 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1217) took 1579 ms. Trace:
I20250109 20:53:05.361063 1166 rpcz_store.cc:276] 0109 20:53:03.781230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.781893 (+ 663us) service_pool.cc:225] Handling call
0109 20:53:05.360856 (+1578963us) ts_tablet_manager.cc:1145] Deleting tablet c6e4cfb5a07841828e5206d100b4f835
0109 20:53:05.360866 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.360956 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1578756}
I20250109 20:53:05.361248 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 589325d5923d40eda5741ff812392de2 (DEFAULT_TABLE table=default.Table_9630bc5dec99481588b309f93dccb0f6 [id=d5771e8ef6fc44ad8b49d580b8169463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.361311 1166 tablet_replica.cc:331] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:05.361496 1166 raft_consensus.cc:2238] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:05.361647 1166 raft_consensus.cc:2267] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:05.361621 1016 tablet_service.cc:1467] Processing CreateTablet for tablet b7fda1ca6d334592af751965338323e9 (DEFAULT_TABLE table=default.Table_5bc430ff7d0847e2a9bff5aa2c007f12 [id=c89f25fe151d42a2bfab6e87226ee003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.362197 1166 ts_tablet_manager.cc:1905] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:05.363780 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6492caf3355c4685a6f0670ad1a352e9 (table default.Table_bc7963618a2e45d6999e681eec0db896 [id=1aa0b264f207481d8c21653dfbbbf547], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6492caf3355c4685a6f0670ad1a352e9 consensus state has no leader
I20250109 20:53:05.364099 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 16077fe2a88d4ee0884701fe0b9a2f1e (DEFAULT_TABLE table=default.Table_fe2e5d6b9d9c4efaa573ac455d4073af [id=c8cf3e8452fc4a0e981414656da5201e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.359254 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb0f2c49c4d4a039121a2f10eb0c6c4 (DEFAULT_TABLE table=default.Table_6d8de56aab614e7a8c51dc1d92c651db [id=4dff8a829f744b59b4114ef0f6141399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.365923 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet de8ea5ea7fa44866b6d7160466f5a896 (table default.Table_64ebceb89d2c4e8291bc096091578453 [id=94d89d19277c4e3fbf4ba420ebe0965c]) successfully deleted
W20250109 20:53:05.366199 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c6e4cfb5a07841828e5206d100b4f835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6e4cfb5a07841828e5206d100b4f835 already in progress: creating tablet
I20250109 20:53:05.373147 2470 raft_consensus.cc:491] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.373273 2470 raft_consensus.cc:513] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.373862 2470 leader_election.cc:290] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.374591 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce0a941f0c14501a947a5c7e2a844e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.374909 1176 tablet_bootstrap.cc:492] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:05.375041 1176 ts_tablet_manager.cc:1397] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.129s user 0.005s sys 0.001s
I20250109 20:53:05.375336 775 raft_consensus.cc:2463] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:53:05.375861 990 leader_election.cc:304] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:05.376183 1176 raft_consensus.cc:357] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.376434 1176 raft_consensus.cc:383] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.376515 1176 raft_consensus.cc:738] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.376816 1176 consensus_queue.cc:260] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.381989 1158 tablet_bootstrap.cc:492] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.382125 1158 ts_tablet_manager.cc:1397] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.009s
I20250109 20:53:05.382759 1158 raft_consensus.cc:357] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.382946 1158 raft_consensus.cc:383] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.382999 1176 ts_tablet_manager.cc:1428] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250109 20:53:05.383009 1158 raft_consensus.cc:738] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.383251 1176 tablet_bootstrap.cc:492] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:05.383205 1158 consensus_queue.cc:260] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.383641 1158 ts_tablet_manager.cc:1428] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:05.383855 1158 tablet_bootstrap.cc:492] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:05.377355 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0e5377172f478cbefcc978ed2a9e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.384807 2475 raft_consensus.cc:2798] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:05.384894 2475 raft_consensus.cc:491] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:05.384960 2475 raft_consensus.cc:3054] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:05.385289 1158 tablet_bootstrap.cc:654] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.386214 1176 tablet_bootstrap.cc:654] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.384845 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f547fb556564da59df6b3743da0288e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.389592 1158 tablet_bootstrap.cc:492] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.389701 1158 ts_tablet_manager.cc:1397] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:53:05.390023 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589325d5923d40eda5741ff812392de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.390290 1158 raft_consensus.cc:357] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.390398 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ffcbca06534d03ab82c02e16608d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.390425 1158 raft_consensus.cc:383] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.390468 1158 raft_consensus.cc:738] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.390596 1158 consensus_queue.cc:260] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.390763 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fb0f2c49c4d4a039121a2f10eb0c6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.390939 1158 ts_tablet_manager.cc:1428] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:05.391173 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fda1ca6d334592af751965338323e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.391193 1158 tablet_bootstrap.cc:492] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:05.391530 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16077fe2a88d4ee0884701fe0b9a2f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.392834 1158 tablet_bootstrap.cc:654] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.396497 1018 tablet_service.cc:1467] Processing CreateTablet for tablet b5f68bdc796548939df69518a1f5294d (DEFAULT_TABLE table=default.Table_b062364df95c4447b2e99c5af63d535d [id=79c215bde5cc4b0a9341ffcf308564da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.396981 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 187bea4487ea43ee96d6969602ef1b08 (DEFAULT_TABLE table=default.Table_578ca6a59d974fd78cc47a19a75f12da [id=2339a9cc70634a0aae60eeeab7a1e11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.397570 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0b412545bb9f4a6a841be1dce4e28c37 (DEFAULT_TABLE table=default.Table_21df3bf9edb24336a97d09e581e2548b [id=7e16b833179e4dc2b9587789030d7226]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.398142 1019 tablet_service.cc:1467] Processing CreateTablet for tablet bca6602ca0534bf8ab4b47d6e287ee2e (DEFAULT_TABLE table=default.Table_8335c1b94e274c0d8d8e927554aa0618 [id=aa3dcd0fcd7440fe9ccd84dd7b28149a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:05.403069 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f3267402bd54b9bb1438a1e15424b3a consensus state has no leader
I20250109 20:53:05.405752 1176 tablet_bootstrap.cc:492] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:05.414415 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6044240c05e41e29eccf27086bdc103. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.416863 1176 ts_tablet_manager.cc:1397] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250109 20:53:05.417616 1176 raft_consensus.cc:357] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.417845 1176 raft_consensus.cc:383] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.417951 1176 raft_consensus.cc:738] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.418232 1176 consensus_queue.cc:260] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.418829 1176 ts_tablet_manager.cc:1428] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:05.419081 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d320aee9b99411eb394d0d28de07ce4" } modify_external_catalogs: true
I20250109 20:53:05.419199 1176 tablet_bootstrap.cc:492] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:05.419247 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d320aee9b99411eb394d0d28de07ce4" } modify_external_catalogs: true
I20250109 20:53:05.420660 1176 tablet_bootstrap.cc:654] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.423390 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c70ddcc69b2b42fcbe133978dd3ac05b
I20250109 20:53:05.423581 2512 tablet.cc:1721] T 7b7dc6809d6547f6b0354b64d7149bf4 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:05.424602 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320] alter complete (version 1)
I20250109 20:53:05.425832 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet c70ddcc69b2b42fcbe133978dd3ac05b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.426088 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 530369cc998a4e9980eed6d4b1f57ad4
I20250109 20:53:05.427863 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 530369cc998a4e9980eed6d4b1f57ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.322288 1165 raft_consensus.cc:357] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.432189 1165 raft_consensus.cc:383] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.432262 1165 raft_consensus.cc:738] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.432456 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1157) took 2096 ms. Trace:
I20250109 20:53:05.432479 1165 consensus_queue.cc:260] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.432520 1160 rpcz_store.cc:276] 0109 20:53:03.335885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.335958 (+ 73us) service_pool.cc:225] Handling call
0109 20:53:05.001573 (+1665615us) ts_tablet_manager.cc:1145] Deleting tablet 06e615ce8e124cfe9432778b6b58e117
0109 20:53:05.014386 (+ 12813us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.096196 (+ 81810us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.319305 (+223109us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.432441 (+113136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665419}
I20250109 20:53:05.432857 1160 tablet_replica.cc:331] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:05.433018 1165 ts_tablet_manager.cc:1428] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.116s user 0.002s sys 0.000s
I20250109 20:53:05.433295 1165 tablet_bootstrap.cc:492] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:05.434924 1165 tablet_bootstrap.cc:654] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.439218 2513 tablet.cc:1721] T 7b7dc6809d6547f6b0354b64d7149bf4 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:05.439307 2469 raft_consensus.cc:491] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.439738 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 06e615ce8e124cfe9432778b6b58e117 (table default.Table_291aa89efe4c406cbebcd12163ceae90 [id=0256b60a9d99439b894214ef4f704f38]) successfully deleted
I20250109 20:53:05.441897 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0b103425555a457eb855c29c1f95fb1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.362033 895 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb0f2c49c4d4a039121a2f10eb0c6c4 (DEFAULT_TABLE table=default.Table_6d8de56aab614e7a8c51dc1d92c651db [id=4dff8a829f744b59b4114ef0f6141399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.362401 885 tablet_service.cc:1467] Processing CreateTablet for tablet 589325d5923d40eda5741ff812392de2 (DEFAULT_TABLE table=default.Table_9630bc5dec99481588b309f93dccb0f6 [id=d5771e8ef6fc44ad8b49d580b8169463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.362676 888 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0e5377172f478cbefcc978ed2a9e0d (DEFAULT_TABLE table=default.Table_31d3d34c2b5347fbb18b9b7d7e034d5b [id=8bdc8ffeaf5e4930b8782cc4d9d89ce6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:05.442466 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b103425555a457eb855c29c1f95fb1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.362845 892 tablet_service.cc:1467] Processing CreateTablet for tablet 0b412545bb9f4a6a841be1dce4e28c37 (DEFAULT_TABLE table=default.Table_21df3bf9edb24336a97d09e581e2548b [id=7e16b833179e4dc2b9587789030d7226]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.363098 887 tablet_service.cc:1467] Processing CreateTablet for tablet b5f68bdc796548939df69518a1f5294d (DEFAULT_TABLE table=default.Table_b062364df95c4447b2e99c5af63d535d [id=79c215bde5cc4b0a9341ffcf308564da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.363361 882 tablet_service.cc:1467] Processing CreateTablet for tablet bca6602ca0534bf8ab4b47d6e287ee2e (DEFAULT_TABLE table=default.Table_8335c1b94e274c0d8d8e927554aa0618 [id=aa3dcd0fcd7440fe9ccd84dd7b28149a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.363582 881 tablet_service.cc:1467] Processing CreateTablet for tablet b7fda1ca6d334592af751965338323e9 (DEFAULT_TABLE table=default.Table_5bc430ff7d0847e2a9bff5aa2c007f12 [id=c89f25fe151d42a2bfab6e87226ee003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.363831 893 tablet_service.cc:1467] Processing CreateTablet for tablet 16077fe2a88d4ee0884701fe0b9a2f1e (DEFAULT_TABLE table=default.Table_fe2e5d6b9d9c4efaa573ac455d4073af [id=c8cf3e8452fc4a0e981414656da5201e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.364117 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3f547fb556564da59df6b3743da0288e (DEFAULT_TABLE table=default.Table_54096fae81dc43028db9aba31cafb2a1 [id=863af0319a3a4e10a512bd0c2a086ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.362676 883 tablet_service.cc:1467] Processing CreateTablet for tablet 187bea4487ea43ee96d6969602ef1b08 (DEFAULT_TABLE table=default.Table_578ca6a59d974fd78cc47a19a75f12da [id=2339a9cc70634a0aae60eeeab7a1e11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.394268 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce0a941f0c14501a947a5c7e2a844e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:05.443342 989 leader_election.cc:343] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.444239 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fb0f2c49c4d4a039121a2f10eb0c6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.445071 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ea3048d16571434f9b5a83dadb1160be" } modify_external_catalogs: true
I20250109 20:53:05.445258 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ea3048d16571434f9b5a83dadb1160be" } modify_external_catalogs: true
I20250109 20:53:05.447973 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_31f7e4437ce94a0eb4369af44e5c466f" } new_table_name: "default.Table_ab628f3ecda348f48bc9442d55bc92b7" modify_external_catalogs: true
I20250109 20:53:05.449321 894 tablet_service.cc:1514] Processing DeleteTablet for tablet cd23012af5574f37944f9e26c07d05d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.450634 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d23131ea9b10461a9b98e2c1c1e743fe" } new_table_name: "default.Table_e7c4991a0b614f52bdb0910bff01b1b2" modify_external_catalogs: true
I20250109 20:53:05.450942 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589325d5923d40eda5741ff812392de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.454363 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8414e40b8fb04357b1f98d7ae514207a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.454883 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8414e40b8fb04357b1f98d7ae514207a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.455765 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0e5377172f478cbefcc978ed2a9e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.457581 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 798aa6062f3447419dbaff60ec445b64
I20250109 20:53:05.459175 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 798aa6062f3447419dbaff60ec445b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.460752 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b412545bb9f4a6a841be1dce4e28c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.464618 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f68bdc796548939df69518a1f5294d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.465411 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bca6602ca0534bf8ab4b47d6e287ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.466216 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fda1ca6d334592af751965338323e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.467366 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16077fe2a88d4ee0884701fe0b9a2f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.468195 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f547fb556564da59df6b3743da0288e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.468976 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187bea4487ea43ee96d6969602ef1b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.471036 2368 raft_consensus.cc:491] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.471185 2368 raft_consensus.cc:513] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.471266 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_45be879858bb4f5c9c9513bec5a7d2ec" } modify_external_catalogs: true
I20250109 20:53:05.471378 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_45be879858bb4f5c9c9513bec5a7d2ec" } modify_external_catalogs: true
I20250109 20:53:05.471814 2368 leader_election.cc:290] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.472772 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6cf525cb44c428f7a39a25c4e62dc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:05.473304 989 leader_election.cc:343] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.475961 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6cf525cb44c428f7a39a25c4e62dc" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:05.476444 990 leader_election.cc:343] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.476558 990 leader_election.cc:304] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.477010 2368 raft_consensus.cc:2743] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.478724 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_71981989b778449aa3784479963a2e24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.479319 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71981989b778449aa3784479963a2e24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.480571 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_321a0a39379a4930b7e2568f85df4bc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.481034 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_321a0a39379a4930b7e2568f85df4bc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.482538 2475 raft_consensus.cc:513] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.483155 2475 leader_election.cc:290] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.483637 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce0a941f0c14501a947a5c7e2a844e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:05.484114 989 leader_election.cc:343] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:05.484375 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0634d1eb0064e9d85b83ea3506cc597 consensus state has no leader
W20250109 20:53:05.484905 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a70be05b5ecd4ce6b37c5b781d299e06 has no consensus state
I20250109 20:53:05.485985 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d8654c154dab465d8991910698027c3f
I20250109 20:53:05.487231 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d8654c154dab465d8991910698027c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.492489 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d8dc3eb6434742bd9f61b5989fa3a6c7" } new_table_name: "default.Table_8a0b2eddb0384893be34d4820890256c" modify_external_catalogs: true
I20250109 20:53:05.496472 2469 raft_consensus.cc:513] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.497202 1176 tablet_bootstrap.cc:492] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:05.497372 1176 ts_tablet_manager.cc:1397] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.078s user 0.016s sys 0.012s
I20250109 20:53:05.498449 1176 raft_consensus.cc:357] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.498701 746 tablet_service.cc:1514] Processing DeleteTablet for tablet f9145358b66d489788ec5cb9d4cd68e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.498802 1176 raft_consensus.cc:383] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.498907 1176 raft_consensus.cc:738] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.499272 1176 consensus_queue.cc:260] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.499840 1176 ts_tablet_manager.cc:1428] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:05.500768 1176 tablet_bootstrap.cc:492] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:05.501232 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce0a941f0c14501a947a5c7e2a844e8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:05.501403 778 raft_consensus.cc:3054] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:05.502507 2469 leader_election.cc:290] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.503137 1176 tablet_bootstrap.cc:654] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.503005 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2d88d72eb7b042edbb97806d615eccf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.503537 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d88d72eb7b042edbb97806d615eccf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.505091 894 tablet_service.cc:1514] Processing DeleteTablet for tablet d8654c154dab465d8991910698027c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.508119 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f68bdc796548939df69518a1f5294d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.508347 2469 raft_consensus.cc:491] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.508651 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b412545bb9f4a6a841be1dce4e28c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.508953 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.509117 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187bea4487ea43ee96d6969602ef1b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.509357 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b with delete_type TABLET_DATA_DELETED (replaced by tablet 50039da210df47ab8476e28387c8a802) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.509522 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bca6602ca0534bf8ab4b47d6e287ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.510607 2475 raft_consensus.cc:491] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.510746 2475 raft_consensus.cc:513] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.511430 2475 leader_election.cc:290] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.512292 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6461f153a5994412ba05c3b687963fd1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.508495 2469 raft_consensus.cc:513] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:05.512913 989 leader_election.cc:343] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.514830 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6461f153a5994412ba05c3b687963fd1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:05.515362 990 leader_election.cc:343] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.515532 990 leader_election.cc:304] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.515938 2469 leader_election.cc:290] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.516129 2475 raft_consensus.cc:2743] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.516309 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a2b017841b9d495cb6eaa4f808555b89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.516856 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2b017841b9d495cb6eaa4f808555b89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.520654 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f03d4e679a1c4793bd83cd979acced88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.522193 747 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb0f2c49c4d4a039121a2f10eb0c6c4 (DEFAULT_TABLE table=default.Table_6d8de56aab614e7a8c51dc1d92c651db [id=4dff8a829f744b59b4114ef0f6141399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.523773 2413 raft_consensus.cc:491] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.525678 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_29736471ff5740c6909a38f7df420f22" } new_table_name: "default.Table_392e3abc25504097871da0f886a3f3b3" modify_external_catalogs: true
I20250109 20:53:05.394838 886 tablet_service.cc:1467] Processing CreateTablet for tablet 33ffcbca06534d03ab82c02e16608d66 (DEFAULT_TABLE table=default.Table_7768e94e2bff455eb490b7cd4e6f1fbb [id=e4d720450c6141fb82420e40b69c13ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.529129 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ffcbca06534d03ab82c02e16608d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.531031 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (table default.Table_159068abc4d842c693e5ae2323d1147d [id=15c9f7d5d3ff4fa4865b55c5892421fb]). Expected version 3 got 0
I20250109 20:53:05.531683 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet ee208bad76734a8fa1ada438bdbbe48c (table default.Table_79fb13be9f0440c9991b67d641ab0b6a [id=238aa3a3a8e74dfba23e60c7d6f049cf]). Expected version 1 got 0
I20250109 20:53:05.425529 884 tablet_service.cc:1514] Processing DeleteTablet for tablet c70ddcc69b2b42fcbe133978dd3ac05b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.427088 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 530369cc998a4e9980eed6d4b1f57ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.533371 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0e5377172f478cbefcc978ed2a9e0d (DEFAULT_TABLE table=default.Table_31d3d34c2b5347fbb18b9b7d7e034d5b [id=8bdc8ffeaf5e4930b8782cc4d9d89ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.534024 884 tablet_service.cc:1467] Processing CreateTablet for tablet 41f0935f17714ecead18b813ccf38f74 (DEFAULT_TABLE table=default.Table_cea365f9151b49bd829baf4014653bd9 [id=3320cb8ebf5641d5ac1faf9082fb0cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.534360 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41f0935f17714ecead18b813ccf38f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.534389 893 tablet_service.cc:1467] Processing CreateTablet for tablet d7aad88078264ecaa7909681261ed132 (DEFAULT_TABLE table=default.Table_48be3078038a4e47a7ece82d379d0c41 [id=1e280512b1a34d24b9c86ef9abfb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.534749 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7aad88078264ecaa7909681261ed132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.535084 891 tablet_service.cc:1467] Processing CreateTablet for tablet 0048433d4a9b4f70b7f4720106390ced (DEFAULT_TABLE table=default.Table_22859e47f7314012b11a0b937538dd3b [id=5763f3e058574ce7bba1332f2f8e5d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.535403 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0048433d4a9b4f70b7f4720106390ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.545346 894 tablet_service.cc:1467] Processing CreateTablet for tablet af4d46c4dbee437cbcd7c0a53997a445 (DEFAULT_TABLE table=default.Table_a5ce359456d84af3baf8a6fccc5a82cc [id=e678edd091e14ed48f15c1bf451a6ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.545712 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4d46c4dbee437cbcd7c0a53997a445. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.547359 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7afdde7589b94916838632c88bc43951"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.547789 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7afdde7589b94916838632c88bc43951 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.551302 883 tablet_service.cc:1467] Processing CreateTablet for tablet 249e612a003545e792ecfd6fc001cbfd (DEFAULT_TABLE table=default.Table_f493c72deafc46099ed5a76527351d35 [id=b76c3f9b30674ae1abc52f9c92726b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.552487 886 tablet_service.cc:1467] Processing CreateTablet for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 (DEFAULT_TABLE table=default.Table_d476f01f037441babd8bd5d6c4e53bb2 [id=e07c66dd551842eb8d6218e823fa466a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.553113 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249e612a003545e792ecfd6fc001cbfd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:05.553248 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 204ae923b8354fc791b5c4fb7197dc76 consensus state has no leader
I20250109 20:53:05.555831 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 798aa6062f3447419dbaff60ec445b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.556129 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533ac2bcf30d4ff091c5c0d5e418fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.563534 1157 ts_tablet_manager.cc:1918] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:05.563661 1157 log.cc:1198] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/bba303032679462c8b16f706096b6d4d
I20250109 20:53:05.565346 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9719bbee9dbc4934992b47bf7b9bd249 (DEFAULT_TABLE table=default.Table_e8e4034056dd4856a0bdb4c83d34b6d9 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:05.565791 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a410e84b795a4e66a554e632c56f3b43 consensus state has no leader
I20250109 20:53:05.566160 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1e66b7acae984dd6911c3fbf1a06780d
I20250109 20:53:05.566799 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e66b7acae984dd6911c3fbf1a06780d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.570511 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0e5377172f478cbefcc978ed2a9e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.572921 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fb0f2c49c4d4a039121a2f10eb0c6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.573170 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_084b21c694b240519af448f7c932a57f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.573716 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_084b21c694b240519af448f7c932a57f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.574162 884 tablet_service.cc:1467] Processing CreateTablet for tablet d47c86f9244c453796f474cf1933e113 (DEFAULT_TABLE table=default.Table_4d2d7ac2ad254b55a416b717dfd03e11 [id=d0025b509ac04785b5548d915e0f2ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.575203 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9719bbee9dbc4934992b47bf7b9bd249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.578485 1157 ts_tablet_manager.cc:1939] T bba303032679462c8b16f706096b6d4d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:05.579092 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3f547fb556564da59df6b3743da0288e (DEFAULT_TABLE table=default.Table_54096fae81dc43028db9aba31cafb2a1 [id=863af0319a3a4e10a512bd0c2a086ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.579680 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f547fb556564da59df6b3743da0288e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.584990 2413 raft_consensus.cc:513] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.587304 748 tablet_service.cc:1467] Processing CreateTablet for tablet 16077fe2a88d4ee0884701fe0b9a2f1e (DEFAULT_TABLE table=default.Table_fe2e5d6b9d9c4efaa573ac455d4073af [id=c8cf3e8452fc4a0e981414656da5201e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.587687 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16077fe2a88d4ee0884701fe0b9a2f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.588627 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 0048433d4a9b4f70b7f4720106390ced (DEFAULT_TABLE table=default.Table_22859e47f7314012b11a0b937538dd3b [id=5763f3e058574ce7bba1332f2f8e5d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.590698 2368 raft_consensus.cc:491] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.589946 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47c86f9244c453796f474cf1933e113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.593789 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a6cf525cb44c428f7a39a25c4e62dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.594621 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 41f0935f17714ecead18b813ccf38f74 (DEFAULT_TABLE table=default.Table_cea365f9151b49bd829baf4014653bd9 [id=3320cb8ebf5641d5ac1faf9082fb0cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.595088 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d7aad88078264ecaa7909681261ed132 (DEFAULT_TABLE table=default.Table_48be3078038a4e47a7ece82d379d0c41 [id=1e280512b1a34d24b9c86ef9abfb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.595217 1025 tablet_service.cc:1467] Processing CreateTablet for tablet af4d46c4dbee437cbcd7c0a53997a445 (DEFAULT_TABLE table=default.Table_a5ce359456d84af3baf8a6fccc5a82cc [id=e678edd091e14ed48f15c1bf451a6ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.595477 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:05.596343 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e66b7acae984dd6911c3fbf1a06780d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.596938 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 (DEFAULT_TABLE table=default.Table_d476f01f037441babd8bd5d6c4e53bb2 [id=e07c66dd551842eb8d6218e823fa466a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.597222 1027 tablet_service.cc:1467] Processing CreateTablet for tablet d47c86f9244c453796f474cf1933e113 (DEFAULT_TABLE table=default.Table_4d2d7ac2ad254b55a416b717dfd03e11 [id=d0025b509ac04785b5548d915e0f2ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.597397 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 249e612a003545e792ecfd6fc001cbfd (DEFAULT_TABLE table=default.Table_f493c72deafc46099ed5a76527351d35 [id=b76c3f9b30674ae1abc52f9c92726b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.597681 2368 raft_consensus.cc:513] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.598280 2368 leader_election.cc:290] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.596653 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 9719bbee9dbc4934992b47bf7b9bd249 (DEFAULT_TABLE table=default.Table_e8e4034056dd4856a0bdb4c83d34b6d9 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.598925 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.600603 2475 raft_consensus.cc:491] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.600759 2475 raft_consensus.cc:513] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.601226 2475 leader_election.cc:290] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:53:05.601393 990 leader_election.cc:343] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.602200 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3c276fab6a4d068d64ea5d23c9a595" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.602365 775 raft_consensus.cc:2463] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:53:05.605154 745 tablet_service.cc:1467] Processing CreateTablet for tablet 589325d5923d40eda5741ff812392de2 (DEFAULT_TABLE table=default.Table_9630bc5dec99481588b309f93dccb0f6 [id=d5771e8ef6fc44ad8b49d580b8169463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.605341 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_392e3abc25504097871da0f886a3f3b3" } new_table_name: "default.Table_46519a61b6b6401a98012df77374a3ea" modify_external_catalogs: true
I20250109 20:53:05.605563 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589325d5923d40eda5741ff812392de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.606153 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3c276fab6a4d068d64ea5d23c9a595" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.607928 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b545e21699b5465eb751e33e57a1fe88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.608035 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:05.608412 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b545e21699b5465eb751e33e57a1fe88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.611995 757 tablet_service.cc:1467] Processing CreateTablet for tablet bca6602ca0534bf8ab4b47d6e287ee2e (DEFAULT_TABLE table=default.Table_8335c1b94e274c0d8d8e927554aa0618 [id=aa3dcd0fcd7440fe9ccd84dd7b28149a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.612035 756 tablet_service.cc:1467] Processing CreateTablet for tablet 0b412545bb9f4a6a841be1dce4e28c37 (DEFAULT_TABLE table=default.Table_21df3bf9edb24336a97d09e581e2548b [id=7e16b833179e4dc2b9587789030d7226]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.612352 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b412545bb9f4a6a841be1dce4e28c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.612514 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bca6602ca0534bf8ab4b47d6e287ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.615407 751 tablet_service.cc:1467] Processing CreateTablet for tablet b5f68bdc796548939df69518a1f5294d (DEFAULT_TABLE table=default.Table_b062364df95c4447b2e99c5af63d535d [id=79c215bde5cc4b0a9341ffcf308564da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.615746 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f68bdc796548939df69518a1f5294d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.616829 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2efcc2a6ec4b109845d8087491ca9f with delete_type TABLET_DATA_DELETED (replaced by tablet 3fc68008840d4b8fae5185718ab454fa) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.621160 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 530369cc998a4e9980eed6d4b1f57ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.621558 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 798aa6062f3447419dbaff60ec445b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.621927 750 tablet_service.cc:1514] Processing DeleteTablet for tablet d8654c154dab465d8991910698027c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.622476 757 tablet_service.cc:1467] Processing CreateTablet for tablet b7fda1ca6d334592af751965338323e9 (DEFAULT_TABLE table=default.Table_5bc430ff7d0847e2a9bff5aa2c007f12 [id=c89f25fe151d42a2bfab6e87226ee003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.622850 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fda1ca6d334592af751965338323e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.622957 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c70ddcc69b2b42fcbe133978dd3ac05b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.626197 745 tablet_service.cc:1467] Processing CreateTablet for tablet 187bea4487ea43ee96d6969602ef1b08 (DEFAULT_TABLE table=default.Table_578ca6a59d974fd78cc47a19a75f12da [id=2339a9cc70634a0aae60eeeab7a1e11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.626609 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187bea4487ea43ee96d6969602ef1b08. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:05.627061 713 leader_election.cc:343] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2ed7abf23ce04a268a460d8e3e97d56b
I20250109 20:53:05.627207 750 tablet_service.cc:1467] Processing CreateTablet for tablet 0048433d4a9b4f70b7f4720106390ced (DEFAULT_TABLE table=default.Table_22859e47f7314012b11a0b937538dd3b [id=5763f3e058574ce7bba1332f2f8e5d52]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:05.627308 713 leader_election.cc:343] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: f03d4e679a1c4793bd83cd979acced88
W20250109 20:53:05.627907 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a410e84b795a4e66a554e632c56f3b43 consensus state has no leader
I20250109 20:53:05.627883 751 tablet_service.cc:1467] Processing CreateTablet for tablet 41f0935f17714ecead18b813ccf38f74 (DEFAULT_TABLE table=default.Table_cea365f9151b49bd829baf4014653bd9 [id=3320cb8ebf5641d5ac1faf9082fb0cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.627936 753 tablet_service.cc:1467] Processing CreateTablet for tablet d7aad88078264ecaa7909681261ed132 (DEFAULT_TABLE table=default.Table_48be3078038a4e47a7ece82d379d0c41 [id=1e280512b1a34d24b9c86ef9abfb33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.628424 740 tablet_service.cc:1467] Processing CreateTablet for tablet 249e612a003545e792ecfd6fc001cbfd (DEFAULT_TABLE table=default.Table_f493c72deafc46099ed5a76527351d35 [id=b76c3f9b30674ae1abc52f9c92726b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.628497 755 tablet_service.cc:1467] Processing CreateTablet for tablet d47c86f9244c453796f474cf1933e113 (DEFAULT_TABLE table=default.Table_4d2d7ac2ad254b55a416b717dfd03e11 [id=d0025b509ac04785b5548d915e0f2ceb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:05.628669 989 leader_election.cc:343] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:05.628990 989 leader_election.cc:343] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.629113 989 leader_election.cc:304] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.629245 743 tablet_service.cc:1467] Processing CreateTablet for tablet 9719bbee9dbc4934992b47bf7b9bd249 (DEFAULT_TABLE table=default.Table_e8e4034056dd4856a0bdb4c83d34b6d9 [id=cfe148d699c24f25bcd068a1a0fb0159]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:05.630399 989 consensus_peers.cc:487] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:05.632369 2488 raft_consensus.cc:491] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.632839 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet e090f72eb1e046319b825ee5fe126332 (table default.Table_e9297199b4064b5f889530ac2757c3ad [id=53929cac72e44474a2dbc7ae26524da3]) successfully deleted
W20250109 20:53:05.633087 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c6e4cfb5a07841828e5206d100b4f835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6e4cfb5a07841828e5206d100b4f835 already in progress: creating tablet
I20250109 20:53:05.633273 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 80da78917f0b4931ab1705371c1fe5dd (table default.Table_500eb283bc7f4131b62740a5e293b785 [id=16cb6983d685473c8d7aba4e39573aae]) successfully deleted
W20250109 20:53:05.633452 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7b8344dbb1c540abadd22531e19a52d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b8344dbb1c540abadd22531e19a52d1 already in progress: creating tablet
W20250109 20:53:05.633610 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6c4abc19297e4029b1b7c5396b49b967 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c4abc19297e4029b1b7c5396b49b967 already in progress: creating tablet
I20250109 20:53:05.633769 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 8f420b7e623243e093d26c95276c5af9 (table default.Table_bb47b9b8617f4706aeb895a16ded22ee [id=ec17429d6df548118b1e53e77bac80c0]) successfully deleted
W20250109 20:53:05.633935 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f0d09d4894634825ad3aa7933bfb6010 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0d09d4894634825ad3aa7933bfb6010 already in progress: creating tablet
W20250109 20:53:05.634093 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21dbbfa0af5b4f0487dd5d9db8efaeda already in progress: creating tablet
I20250109 20:53:05.634234 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet b68d23fb2df743f4afa2682f6d2e3efa (table default.Table_07f19c8f31bb48c598a7a2b8dc7187af [id=e0c7ea945a0a40e0a4a3e369c7455eca]) successfully deleted
W20250109 20:53:05.634424 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1faf5d79a9b43ffafc9c34a5937fcc0 already in progress: creating tablet
W20250109 20:53:05.635255 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8f2ba81d736745fb9d56fd9a7c229153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f2ba81d736745fb9d56fd9a7c229153 already in progress: creating tablet
I20250109 20:53:05.637117 1160 raft_consensus.cc:2238] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:05.637360 1160 raft_consensus.cc:2267] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:05.637493 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_16df12cac132428ca8cc7cac50d05578" } modify_external_catalogs: true
I20250109 20:53:05.637643 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_16df12cac132428ca8cc7cac50d05578" } modify_external_catalogs: true
I20250109 20:53:05.638005 1160 ts_tablet_manager.cc:1905] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:05.638077 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0c702c1df34a466da3d2c24f0150e438" } modify_external_catalogs: true
I20250109 20:53:05.638201 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0c702c1df34a466da3d2c24f0150e438" } modify_external_catalogs: true
I20250109 20:53:05.638844 2475 raft_consensus.cc:2743] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.639784 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_281a8c43ff2e48dea9d18f9906dcb4f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.640406 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_281a8c43ff2e48dea9d18f9906dcb4f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.640672 2469 raft_consensus.cc:491] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.640843 2469 raft_consensus.cc:513] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.642597 1166 ts_tablet_manager.cc:1918] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:05.642726 1166 log.cc:1198] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ea16fcc30f2c4b2eac53f20fcebd043f
I20250109 20:53:05.643069 1166 ts_tablet_manager.cc:1939] T ea16fcc30f2c4b2eac53f20fcebd043f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:05.643533 778 raft_consensus.cc:2463] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:05.644075 990 leader_election.cc:304] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.644320 2368 raft_consensus.cc:491] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.644439 2368 raft_consensus.cc:513] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.644990 2368 leader_election.cc:290] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.645116 2368 raft_consensus.cc:2798] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:05.645210 2368 raft_consensus.cc:695] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:53:05.645330 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7aad88078264ecaa7909681261ed132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.645514 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0048433d4a9b4f70b7f4720106390ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.645442 2368 consensus_queue.cc:237] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.647090 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47c86f9244c453796f474cf1933e113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.647238 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249e612a003545e792ecfd6fc001cbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.648581 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9719bbee9dbc4934992b47bf7b9bd249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.648507 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.648931 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41f0935f17714ecead18b813ccf38f74. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:05.649314 989 leader_election.cc:343] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.650789 1165 tablet_bootstrap.cc:492] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:05.650977 1165 ts_tablet_manager.cc:1397] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.218s user 0.011s sys 0.000s
I20250109 20:53:05.651784 1165 raft_consensus.cc:357] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.652035 1165 raft_consensus.cc:383] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.652170 1165 raft_consensus.cc:738] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.652601 1165 consensus_queue.cc:260] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.658762 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0048433d4a9b4f70b7f4720106390ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.665110 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7aad88078264ecaa7909681261ed132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.667326 1165 ts_tablet_manager.cc:1428] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250109 20:53:05.668614 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_b2c8d2b0a7794d608e3a0f1aab3c0e77 [id=f2abcc91ea094f11a4d434ae57e6f404] alter complete (version 1)
I20250109 20:53:05.669101 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4d46c4dbee437cbcd7c0a53997a445. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.670545 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c075a9e071d84c4ca107998bcdc8af22" } modify_external_catalogs: true
I20250109 20:53:05.670722 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c075a9e071d84c4ca107998bcdc8af22" } modify_external_catalogs: true
I20250109 20:53:05.673332 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533ac2bcf30d4ff091c5c0d5e418fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.676045 741 tablet_service.cc:1467] Processing CreateTablet for tablet 33ffcbca06534d03ab82c02e16608d66 (DEFAULT_TABLE table=default.Table_7768e94e2bff455eb490b7cd4e6f1fbb [id=e4d720450c6141fb82420e40b69c13ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.677289 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47c86f9244c453796f474cf1933e113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.677904 883 tablet_service.cc:1467] Processing CreateTablet for tablet a70be05b5ecd4ce6b37c5b781d299e06 (DEFAULT_TABLE table=default.Table_ab628f3ecda348f48bc9442d55bc92b7 [id=06c0fa602d2b48fdb3aee6767661442e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.678738 887 tablet_service.cc:1467] Processing CreateTablet for tablet 068f97bb5a264e539f73cdd415a0e1ab (DEFAULT_TABLE table=default.Table_d1a93305dc21489bb4cc1bb5f36ed408 [id=d450c49daa254946892bf7a345f84825]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.679194 881 tablet_service.cc:1467] Processing CreateTablet for tablet ad7738078d5040e08a1b9ba8ae1ff43c (DEFAULT_TABLE table=default.Table_1a37a751353247b3a261c2adc68a6034 [id=f8eae9f03f2d4682b7a49e31ac1b566d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.681298 886 tablet_service.cc:1467] Processing CreateTablet for tablet a1ed66fe3b46456ea784a3471575c052 (DEFAULT_TABLE table=default.Table_8414e40b8fb04357b1f98d7ae514207a [id=26228514302e4d199e778c71b1b063d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.681797 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249e612a003545e792ecfd6fc001cbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.682328 884 tablet_service.cc:1467] Processing CreateTablet for tablet d914848ad47142f4b64597fd00abdacd (DEFAULT_TABLE table=default.Table_0b103425555a457eb855c29c1f95fb1c [id=837625c2a8e74f8d85e5cfc9a3298a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.683080 1165 tablet_bootstrap.cc:492] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:05.632486 2488 raft_consensus.cc:513] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.684871 1176 tablet_bootstrap.cc:492] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:05.685012 1176 ts_tablet_manager.cc:1397] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.185s user 0.015s sys 0.019s
I20250109 20:53:05.685300 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9719bbee9dbc4934992b47bf7b9bd249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.685794 1176 raft_consensus.cc:357] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.685978 1176 raft_consensus.cc:383] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.686071 1176 raft_consensus.cc:738] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.686302 1176 consensus_queue.cc:260] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.686962 1176 ts_tablet_manager.cc:1428] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:05.687254 1176 tablet_bootstrap.cc:492] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:05.687769 1165 tablet_bootstrap.cc:654] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.688867 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41f0935f17714ecead18b813ccf38f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.689885 1176 tablet_bootstrap.cc:654] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:05.689847 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_42be9454af31421f84024e4849874065" } new_table_name: "default.Table_784476cd98e541bfa7746bf21d81c980" modify_external_catalogs: true
I20250109 20:53:05.691877 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4c73eadb1a204cbdb9d845b96cd2c4b1" } new_table_name: "default.Table_773def38f29b4a5c9c413cbf28623c4d" modify_external_catalogs: true
I20250109 20:53:05.694051 446 catalog_manager.cc:5526] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
I20250109 20:53:05.694487 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:05.694602 2368 raft_consensus.cc:491] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.694718 2368 raft_consensus.cc:513] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.695314 2368 leader_election.cc:290] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.695977 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deedac68322e460b8d62adbe849de3f2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:05.696755 989 leader_election.cc:343] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.698112 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3fc68008840d4b8fae5185718ab454fa
I20250109 20:53:05.698247 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_98aa779b222a4fa1bb03d9c1d84dc2bf" } modify_external_catalogs: true
I20250109 20:53:05.698380 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_98aa779b222a4fa1bb03d9c1d84dc2bf" } modify_external_catalogs: true
I20250109 20:53:05.699079 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d914848ad47142f4b64597fd00abdacd (DEFAULT_TABLE table=default.Table_0b103425555a457eb855c29c1f95fb1c [id=837625c2a8e74f8d85e5cfc9a3298a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.699316 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5ae81d22bb904903a75590beca4b7ee2
I20250109 20:53:05.699575 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 068f97bb5a264e539f73cdd415a0e1ab (DEFAULT_TABLE table=default.Table_d1a93305dc21489bb4cc1bb5f36ed408 [id=d450c49daa254946892bf7a345f84825]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.700172 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc68008840d4b8fae5185718ab454fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.700160 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a70be05b5ecd4ce6b37c5b781d299e06 (DEFAULT_TABLE table=default.Table_ab628f3ecda348f48bc9442d55bc92b7 [id=06c0fa602d2b48fdb3aee6767661442e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.700348 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae81d22bb904903a75590beca4b7ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.700577 1034 tablet_service.cc:1467] Processing CreateTablet for tablet ad7738078d5040e08a1b9ba8ae1ff43c (DEFAULT_TABLE table=default.Table_1a37a751353247b3a261c2adc68a6034 [id=f8eae9f03f2d4682b7a49e31ac1b566d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.700608 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_40809af1edea441a8e0c9402b04b0c74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.701022 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40809af1edea441a8e0c9402b04b0c74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.701136 1020 tablet_service.cc:1467] Processing CreateTablet for tablet a1ed66fe3b46456ea784a3471575c052 (DEFAULT_TABLE table=default.Table_8414e40b8fb04357b1f98d7ae514207a [id=26228514302e4d199e778c71b1b063d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.703677 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc68008840d4b8fae5185718ab454fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.703708 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae81d22bb904903a75590beca4b7ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.707837 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d30d928c44a84263ad93d9ee36b2d632
W20250109 20:53:05.707928 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet affb5b43f6314e08b0deb3b2e8b40207 consensus state has no leader
I20250109 20:53:05.708766 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d30d928c44a84263ad93d9ee36b2d632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:05.709177 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44af81df98ae4b05927dab7b4246de9f consensus state has no leader
I20250109 20:53:05.710949 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d914848ad47142f4b64597fd00abdacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.711539 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad7738078d5040e08a1b9ba8ae1ff43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.711993 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068f97bb5a264e539f73cdd415a0e1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.712450 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70be05b5ecd4ce6b37c5b781d299e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.712953 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ed66fe3b46456ea784a3471575c052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.715801 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d5e93f3b4c36485a92cfe28d5156b146
I20250109 20:53:05.716672 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e93f3b4c36485a92cfe28d5156b146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.737035 2368 raft_consensus.cc:491] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.737167 2368 raft_consensus.cc:513] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.737742 2368 leader_election.cc:290] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.738988 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e56f30e6f3450a8426cd40db89e009 with delete_type TABLET_DATA_DELETED (replaced by tablet eaea4f239cce41a997496a1c94810ac9) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.739548 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1218) took 1932 ms. Trace:
I20250109 20:53:05.739650 1166 rpcz_store.cc:276] 0109 20:53:03.807237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.809843 (+ 2606us) service_pool.cc:225] Handling call
0109 20:53:05.361271 (+1551428us) ts_tablet_manager.cc:1145] Deleting tablet ea16fcc30f2c4b2eac53f20fcebd043f
0109 20:53:05.361280 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.478439 (+117159us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.642482 (+164043us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.739527 (+ 97045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1551242}
I20250109 20:53:05.739980 1166 tablet_replica.cc:331] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:05.740163 1166 raft_consensus.cc:2238] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:05.740309 1166 raft_consensus.cc:2267] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:05.740809 1166 ts_tablet_manager.cc:1905] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:05.741050 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2efcc2a6ec4b109845d8087491ca9f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.741248 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2414450f78c54a5da31248d94e856cbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.741477 2032 raft_consensus.cc:491] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.741596 2032 raft_consensus.cc:513] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:05.741727 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2414450f78c54a5da31248d94e856cbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.742208 2032 leader_election.cc:290] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:05.743295 854 leader_election.cc:343] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: daac99da5d10475c9de263bef27baeb7
I20250109 20:53:05.743944 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ea16fcc30f2c4b2eac53f20fcebd043f (table default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d [id=35315fe696f042ee8700dd905bce5c5e]) successfully deleted
I20250109 20:53:05.744535 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet d30d928c44a84263ad93d9ee36b2d632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:05.744683 989 leader_election.cc:343] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.753800 1101 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:05.758345 1176 tablet_bootstrap.cc:492] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:05.758464 1176 ts_tablet_manager.cc:1397] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.001s
I20250109 20:53:05.758796 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcb6780422541c89343e98e02938d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.759052 1176 raft_consensus.cc:357] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.759202 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e93f3b4c36485a92cfe28d5156b146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.759236 1176 raft_consensus.cc:383] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.759287 1176 raft_consensus.cc:738] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.759462 1176 consensus_queue.cc:260] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.759897 1176 ts_tablet_manager.cc:1428] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:05.760138 1176 tablet_bootstrap.cc:492] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:05.761247 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6da66d9b734d4dfc848a5ae9e5e184dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.761289 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_07fd41bf19254a88bc4d0169c627b3c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.761770 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6da66d9b734d4dfc848a5ae9e5e184dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.762843 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e1e70e4ec454bb19ce9f47d5272e0e0" } new_table_name: "default.Table_8a88793fc39041e6b8df77f8d2f9f663" modify_external_catalogs: true
I20250109 20:53:05.763517 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_db4cd004494744f892fc049ed4e65a4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.763899 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db4cd004494744f892fc049ed4e65a4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.764649 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1257) took 1736 ms. Trace:
I20250109 20:53:05.764761 1157 rpcz_store.cc:276] 0109 20:53:04.027741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.027792 (+ 51us) service_pool.cc:225] Handling call
0109 20:53:05.151638 (+1123846us) ts_tablet_manager.cc:1145] Deleting tablet bba303032679462c8b16f706096b6d4d
0109 20:53:05.151645 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.155235 (+ 3590us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.563424 (+408189us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.764620 (+201196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123665}
I20250109 20:53:05.766606 2469 leader_election.cc:290] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.768985 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c271524e66094f2e9f698e6e174de073" } new_table_name: "default.Table_c644cd1f69bd4093828ae257b73ccb1d" modify_external_catalogs: true
I20250109 20:53:05.769378 2413 leader_election.cc:290] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.769919 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ec489233361c4a81bb3bdc61530eb072" } modify_external_catalogs: true
I20250109 20:53:05.770056 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ec489233361c4a81bb3bdc61530eb072" } modify_external_catalogs: true
I20250109 20:53:05.770704 756 tablet_service.cc:1467] Processing CreateTablet for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 (DEFAULT_TABLE table=default.Table_d476f01f037441babd8bd5d6c4e53bb2 [id=e07c66dd551842eb8d6218e823fa466a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.771313 2488 leader_election.cc:290] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.771610 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533ac2bcf30d4ff091c5c0d5e418fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.771885 757 tablet_service.cc:1467] Processing CreateTablet for tablet af4d46c4dbee437cbcd7c0a53997a445 (DEFAULT_TABLE table=default.Table_a5ce359456d84af3baf8a6fccc5a82cc [id=e678edd091e14ed48f15c1bf451a6ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.772173 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4d46c4dbee437cbcd7c0a53997a445. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:05.761775 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07fd41bf19254a88bc4d0169c627b3c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.772701 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e66b7acae984dd6911c3fbf1a06780d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.772915 745 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a8bbb3da84088a98c816102e4e9df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.764915 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet face19003d0042669e3f22553f9fe19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.773517 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet bba303032679462c8b16f706096b6d4d (table default.Table_eb7bd4795ee740f0ba9088dc52e7b383 [id=6a25ad9a50db4cd7b62f73b4e268fdae]) successfully deleted
I20250109 20:53:05.773768 1157 tablet_replica.cc:331] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:05.773973 1157 raft_consensus.cc:2238] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:05.774096 1157 raft_consensus.cc:2267] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:05.774555 1157 ts_tablet_manager.cc:1905] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:05.779011 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_21a5abc833a749e487f214e77ffb3938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.779403 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21a5abc833a749e487f214e77ffb3938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:05.781179 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 563c1fcad384470a9f7663d48ad52312 consensus state has no leader
W20250109 20:53:05.781319 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3c2769694f14ddd8d2e351974949b77 consensus state has no leader
I20250109 20:53:05.784619 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_591e8768478a4bd28f197e2b94197c39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.785043 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_591e8768478a4bd28f197e2b94197c39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.784976 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_190f7022156a45b0b2905f93a7f274df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.785351 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_190f7022156a45b0b2905f93a7f274df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.788938 2368 raft_consensus.cc:491] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.789076 2368 raft_consensus.cc:513] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.789714 2368 leader_election.cc:290] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.790369 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc54e97e91c54afea098624bd85fcc6e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:05.790964 989 leader_election.cc:343] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.794144 2368 raft_consensus.cc:491] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.794260 2368 raft_consensus.cc:513] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.794781 2368 leader_election.cc:290] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:05.795261 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b4c038573a4afa879ecbcba75c35a7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:05.795778 989 leader_election.cc:343] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.770911 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deedac68322e460b8d62adbe849de3f2" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.805044 1158 tablet_bootstrap.cc:492] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:05.805186 1158 ts_tablet_manager.cc:1397] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.414s user 0.002s sys 0.014s
I20250109 20:53:05.805135 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.806591 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2efcc2a6ec4b109845d8087491ca9f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:05.806800 990 leader_election.cc:343] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.807054 990 leader_election.cc:304] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.807328 2368 raft_consensus.cc:2743] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:05.807492 990 leader_election.cc:343] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.807595 990 leader_election.cc:304] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.807788 2368 raft_consensus.cc:2743] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.812389 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc54e97e91c54afea098624bd85fcc6e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:05.812839 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ed66fe3b46456ea784a3471575c052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.813236 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d914848ad47142f4b64597fd00abdacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.813916 746 tablet_service.cc:1467] Processing CreateTablet for tablet d914848ad47142f4b64597fd00abdacd (DEFAULT_TABLE table=default.Table_0b103425555a457eb855c29c1f95fb1c [id=837625c2a8e74f8d85e5cfc9a3298a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.815771 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad7738078d5040e08a1b9ba8ae1ff43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.819236 2032 raft_consensus.cc:491] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.819379 2032 raft_consensus.cc:513] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.819973 2032 leader_election.cc:290] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.820158 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068f97bb5a264e539f73cdd415a0e1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.824956 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70be05b5ecd4ce6b37c5b781d299e06. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:05.831857 854 leader_election.cc:343] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 728e9b6101e3447fb3c24b3917aa4872
I20250109 20:53:05.832171 2032 raft_consensus.cc:491] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.832304 2032 raft_consensus.cc:513] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.833020 2032 leader_election.cc:290] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:05.834316 854 leader_election.cc:343] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 57e56f30e6f3450a8426cd40db89e009
I20250109 20:53:05.838781 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9a8af3d5811647918198f9152cac1a8a" } new_table_name: "default.Table_6d59464208774e168bab69b28a04bd5c" modify_external_catalogs: true
I20250109 20:53:05.843058 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 4fe6bd7bc23a4d8095f025b01dd5933c from {username='slave'} at 127.0.0.1:33768
W20250109 20:53:05.846520 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14e483cd4ae64fe1aa31cb0e4b6d6338 consensus state has no leader
I20250109 20:53:05.855090 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33ffcbca06534d03ab82c02e16608d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.855365 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a4bd298f3639422487e396ef57dd2756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:05.855625 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d914848ad47142f4b64597fd00abdacd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:05.855985 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4bd298f3639422487e396ef57dd2756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.860417 886 tablet_service.cc:1467] Processing CreateTablet for tablet 70e96fa48fa649909ef93a70a57c023c (DEFAULT_TABLE table=default.Table_281a8c43ff2e48dea9d18f9906dcb4f2 [id=47cfabdb293b421b8b5a9b9e263d3901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.860836 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e96fa48fa649909ef93a70a57c023c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.863565 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 70e96fa48fa649909ef93a70a57c023c (DEFAULT_TABLE table=default.Table_281a8c43ff2e48dea9d18f9906dcb4f2 [id=47cfabdb293b421b8b5a9b9e263d3901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.864697 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4fe6bd7bc23a4d8095f025b01dd5933c
I20250109 20:53:05.865020 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfa04c33e104c148a73c545284c20a1 (DEFAULT_TABLE table=default.Table_b545e21699b5465eb751e33e57a1fe88 [id=ef90c81c28974be68b6f53132f35170f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.865141 450 catalog_manager.cc:6596] ReplaceTablet: tablet 4fe6bd7bc23a4d8095f025b01dd5933c deleted and replaced by tablet 14ce71f6c3c443a0862658ab3b7510e2
I20250109 20:53:05.865466 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe6bd7bc23a4d8095f025b01dd5933c with delete_type TABLET_DATA_DELETED (replaced by tablet 14ce71f6c3c443a0862658ab3b7510e2) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.866859 1034 tablet_service.cc:1467] Processing CreateTablet for tablet e54e9be016584fde911c4524969d0789 (DEFAULT_TABLE table=default.Table_084b21c694b240519af448f7c932a57f [id=4f2bea00b14340f3a83a4b7c0fd1e23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.869628 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7ca67649b64d26b9bf8341e6d59647 (DEFAULT_TABLE table=default.Table_71981989b778449aa3784479963a2e24 [id=d9a0a1d05896435dab47159f1057b6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.869956 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7ca67649b64d26b9bf8341e6d59647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.871147 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet fb57395c2e684e878baf53d76802dc45
I20250109 20:53:05.871124 884 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfa04c33e104c148a73c545284c20a1 (DEFAULT_TABLE table=default.Table_b545e21699b5465eb751e33e57a1fe88 [id=ef90c81c28974be68b6f53132f35170f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.871408 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bfa04c33e104c148a73c545284c20a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.872785 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7ca67649b64d26b9bf8341e6d59647 (DEFAULT_TABLE table=default.Table_71981989b778449aa3784479963a2e24 [id=d9a0a1d05896435dab47159f1057b6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.873718 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3eb37633d0cd466a9e79e8fa50d54f03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.874143 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3eb37633d0cd466a9e79e8fa50d54f03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.874632 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe6bd7bc23a4d8095f025b01dd5933c with delete_type TABLET_DATA_DELETED (replaced by tablet 14ce71f6c3c443a0862658ab3b7510e2) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.875181 881 tablet_service.cc:1467] Processing CreateTablet for tablet e54e9be016584fde911c4524969d0789 (DEFAULT_TABLE table=default.Table_084b21c694b240519af448f7c932a57f [id=4f2bea00b14340f3a83a4b7c0fd1e23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.875483 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e54e9be016584fde911c4524969d0789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.875483 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet fb57395c2e684e878baf53d76802dc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:05.877861 887 tablet_service.cc:1467] Processing CreateTablet for tablet 452c685f82c94d8da488e15fac44d6b2 (DEFAULT_TABLE table=default.Table_a2b017841b9d495cb6eaa4f808555b89 [id=066d33166d454af591f8c630254335f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.878188 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452c685f82c94d8da488e15fac44d6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.878687 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 452c685f82c94d8da488e15fac44d6b2 (DEFAULT_TABLE table=default.Table_a2b017841b9d495cb6eaa4f808555b89 [id=066d33166d454af591f8c630254335f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.880259 888 tablet_service.cc:1467] Processing CreateTablet for tablet ffba070fd36342369f4122da2b63eb3a (DEFAULT_TABLE table=default.Table_321a0a39379a4930b7e2568f85df4bc8 [id=5a0007dafb014a309d53cff3b6cda912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.880563 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffba070fd36342369f4122da2b63eb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.884980 1019 tablet_service.cc:1467] Processing CreateTablet for tablet ffba070fd36342369f4122da2b63eb3a (DEFAULT_TABLE table=default.Table_321a0a39379a4930b7e2568f85df4bc8 [id=5a0007dafb014a309d53cff3b6cda912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.886078 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 989e420afd094ff5adbe82ed4d75f7c8 (DEFAULT_TABLE table=default.Table_2d88d72eb7b042edbb97806d615eccf2 [id=5f154b25db694a758cf47d0a99e9f69d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.886094 895 tablet_service.cc:1467] Processing CreateTablet for tablet 989e420afd094ff5adbe82ed4d75f7c8 (DEFAULT_TABLE table=default.Table_2d88d72eb7b042edbb97806d615eccf2 [id=5f154b25db694a758cf47d0a99e9f69d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.886456 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989e420afd094ff5adbe82ed4d75f7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.889448 885 tablet_service.cc:1467] Processing CreateTablet for tablet 87e5ff14595f4c9388d3d586ecfefbeb (DEFAULT_TABLE table=default.Table_7afdde7589b94916838632c88bc43951 [id=6c20d97b4623463fbece51dddcccac1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.889735 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e5ff14595f4c9388d3d586ecfefbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.891323 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 87e5ff14595f4c9388d3d586ecfefbeb (DEFAULT_TABLE table=default.Table_7afdde7589b94916838632c88bc43951 [id=6c20d97b4623463fbece51dddcccac1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.904078 883 tablet_service.cc:1514] Processing DeleteTablet for tablet fb57395c2e684e878baf53d76802dc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:05.904960 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7dbdb0f9160444ca9abff301774ac41d" } modify_external_catalogs: true
I20250109 20:53:05.905130 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7dbdb0f9160444ca9abff301774ac41d" } modify_external_catalogs: true
I20250109 20:53:05.908808 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e5d7cc7fe84d481a974efb955fbeb6e4" } modify_external_catalogs: true
I20250109 20:53:05.908962 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e5d7cc7fe84d481a974efb955fbeb6e4" } modify_external_catalogs: true
I20250109 20:53:05.910764 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b4c038573a4afa879ecbcba75c35a7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:05.911164 990 leader_election.cc:343] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.911281 990 leader_election.cc:304] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:05.911665 990 leader_election.cc:343] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.911741 990 leader_election.cc:304] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.912045 2368 raft_consensus.cc:2743] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.912181 2368 raft_consensus.cc:2743] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.912904 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a0874353369a4986b5b3e814940a97a0" } new_table_name: "default.Table_5c07b67a27f940f58b3bfabcabb0a3d8" modify_external_catalogs: true
I20250109 20:53:05.912922 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6d708581ee5c4179af45f5e8ce12bec5" } modify_external_catalogs: true
I20250109 20:53:05.913062 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6d708581ee5c4179af45f5e8ce12bec5" } modify_external_catalogs: true
I20250109 20:53:05.916507 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0309fd73acdf416d94c30a93bc3e1cfe" } new_table_name: "default.Table_814fdd9e9bc94c5196209064664ebcef" modify_external_catalogs: true
I20250109 20:53:05.918907 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b339612e60ae46b48f88dc6d43a1a530" } modify_external_catalogs: true
I20250109 20:53:05.919201 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_94097e767754490fa0d9c7402cc172dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.919601 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94097e767754490fa0d9c7402cc172dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:05.921911 990 leader_election.cc:343] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:05.922014 990 leader_election.cc:304] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:05.922364 2475 raft_consensus.cc:2743] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:05.923120 740 tablet_service.cc:1467] Processing CreateTablet for tablet a70be05b5ecd4ce6b37c5b781d299e06 (DEFAULT_TABLE table=default.Table_ab628f3ecda348f48bc9442d55bc92b7 [id=06c0fa602d2b48fdb3aee6767661442e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.925233 2488 raft_consensus.cc:491] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.925360 2488 raft_consensus.cc:513] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.926038 2488 leader_election.cc:290] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.926833 742 tablet_service.cc:1467] Processing CreateTablet for tablet ad7738078d5040e08a1b9ba8ae1ff43c (DEFAULT_TABLE table=default.Table_1a37a751353247b3a261c2adc68a6034 [id=f8eae9f03f2d4682b7a49e31ac1b566d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.933151 750 tablet_service.cc:1467] Processing CreateTablet for tablet 068f97bb5a264e539f73cdd415a0e1ab (DEFAULT_TABLE table=default.Table_d1a93305dc21489bb4cc1bb5f36ed408 [id=d450c49daa254946892bf7a345f84825]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.934125 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae81d22bb904903a75590beca4b7ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.934952 746 tablet_service.cc:1467] Processing CreateTablet for tablet 70e96fa48fa649909ef93a70a57c023c (DEFAULT_TABLE table=default.Table_281a8c43ff2e48dea9d18f9906dcb4f2 [id=47cfabdb293b421b8b5a9b9e263d3901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.935576 757 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfa04c33e104c148a73c545284c20a1 (DEFAULT_TABLE table=default.Table_b545e21699b5465eb751e33e57a1fe88 [id=ef90c81c28974be68b6f53132f35170f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.936177 739 tablet_service.cc:1514] Processing DeleteTablet for tablet d5e93f3b4c36485a92cfe28d5156b146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.936292 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc68008840d4b8fae5185718ab454fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.933956 744 tablet_service.cc:1467] Processing CreateTablet for tablet a1ed66fe3b46456ea784a3471575c052 (DEFAULT_TABLE table=default.Table_8414e40b8fb04357b1f98d7ae514207a [id=26228514302e4d199e778c71b1b063d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.937544 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4c7ca67649b64d26b9bf8341e6d59647 (DEFAULT_TABLE table=default.Table_71981989b778449aa3784479963a2e24 [id=d9a0a1d05896435dab47159f1057b6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.938774 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed7abf23ce04a268a460d8e3e97d56b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.939404 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d30d928c44a84263ad93d9ee36b2d632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.940505 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03d4e679a1c4793bd83cd979acced88" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.946820 2032 raft_consensus.cc:491] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.946966 2032 raft_consensus.cc:513] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.947603 2032 leader_election.cc:290] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:05.948704 854 leader_election.cc:343] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: ca816507d0c047668a51f6e7ebce18cb
I20250109 20:53:05.948984 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_34d03276cc5d416a85003ea188706357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.949568 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34d03276cc5d416a85003ea188706357 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.949920 1157 ts_tablet_manager.cc:1918] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:05.950078 1157 log.cc:1198] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/225046d45c1c43b488babf5da8de75d7
I20250109 20:53:05.953796 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ed66fe3b46456ea784a3471575c052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.954406 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad7738078d5040e08a1b9ba8ae1ff43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.954632 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068f97bb5a264e539f73cdd415a0e1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.955099 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70be05b5ecd4ce6b37c5b781d299e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.956815 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bfa04c33e104c148a73c545284c20a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.957334 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7ca67649b64d26b9bf8341e6d59647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.939010 1158 raft_consensus.cc:357] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.958287 1158 raft_consensus.cc:383] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:05.958345 1158 raft_consensus.cc:738] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:05.958523 1158 consensus_queue.cc:260] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.959488 745 tablet_service.cc:1467] Processing CreateTablet for tablet 989e420afd094ff5adbe82ed4d75f7c8 (DEFAULT_TABLE table=default.Table_2d88d72eb7b042edbb97806d615eccf2 [id=5f154b25db694a758cf47d0a99e9f69d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.960202 2488 raft_consensus.cc:491] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.960352 2488 raft_consensus.cc:513] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.960997 2488 leader_election.cc:290] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.960220 754 tablet_service.cc:1467] Processing CreateTablet for tablet ffba070fd36342369f4122da2b63eb3a (DEFAULT_TABLE table=default.Table_321a0a39379a4930b7e2568f85df4bc8 [id=5a0007dafb014a309d53cff3b6cda912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.962990 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e96fa48fa649909ef93a70a57c023c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.963622 747 tablet_service.cc:1467] Processing CreateTablet for tablet 87e5ff14595f4c9388d3d586ecfefbeb (DEFAULT_TABLE table=default.Table_7afdde7589b94916838632c88bc43951 [id=6c20d97b4623463fbece51dddcccac1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.963907 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e5ff14595f4c9388d3d586ecfefbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.964874 756 tablet_service.cc:1514] Processing DeleteTablet for tablet fb57395c2e684e878baf53d76802dc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.965291 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989e420afd094ff5adbe82ed4d75f7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.965737 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffba070fd36342369f4122da2b63eb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.977043 758 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe6bd7bc23a4d8095f025b01dd5933c with delete_type TABLET_DATA_DELETED (replaced by tablet 14ce71f6c3c443a0862658ab3b7510e2) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:05.981184 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dce041dd4c504e0694d63c2ada509f81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.981729 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dce041dd4c504e0694d63c2ada509f81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.982050 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0e4aa7de575f45cfbbd9528398ca6aaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:05.982441 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e4aa7de575f45cfbbd9528398ca6aaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:05.982807 1157 ts_tablet_manager.cc:1939] T 225046d45c1c43b488babf5da8de75d7 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:05.986018 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.987614 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724d15e05a864852a118badae13a5d3d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.988449 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8539cefebb844b3098a308ca580b0e92" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.990916 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cede2a1882946a181b18d66f12c5114" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:05.991575 2413 raft_consensus.cc:491] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:05.991709 2413 raft_consensus.cc:513] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:05.992347 2413 leader_election.cc:290] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:05.995155 749 tablet_service.cc:1467] Processing CreateTablet for tablet 452c685f82c94d8da488e15fac44d6b2 (DEFAULT_TABLE table=default.Table_a2b017841b9d495cb6eaa4f808555b89 [id=066d33166d454af591f8c630254335f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:05.995620 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452c685f82c94d8da488e15fac44d6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:05.997849 739 tablet_service.cc:1467] Processing CreateTablet for tablet e54e9be016584fde911c4524969d0789 (DEFAULT_TABLE table=default.Table_084b21c694b240519af448f7c932a57f [id=4f2bea00b14340f3a83a4b7c0fd1e23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.012988 2475 consensus_queue.cc:1035] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:06.017028 1158 ts_tablet_manager.cc:1428] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.212s user 0.001s sys 0.001s
I20250109 20:53:06.017302 1158 tablet_bootstrap.cc:492] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:06.023017 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e54e9be016584fde911c4524969d0789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.027716 1158 tablet_bootstrap.cc:654] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:06.050097 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 396857us
I20250109 20:53:06.087604 1165 tablet_bootstrap.cc:492] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.088186 1165 ts_tablet_manager.cc:1397] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.405s user 0.021s sys 0.040s
I20250109 20:53:06.089322 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1258) took 1982 ms. Trace:
I20250109 20:53:06.089432 1157 rpcz_store.cc:276] 0109 20:53:04.106580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.106630 (+ 50us) service_pool.cc:225] Handling call
0109 20:53:05.765048 (+1658418us) ts_tablet_manager.cc:1145] Deleting tablet 225046d45c1c43b488babf5da8de75d7
0109 20:53:05.773725 (+ 8677us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.842321 (+ 68596us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:05.949788 (+107467us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.089297 (+139509us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35221,"tablet-delete.queue_time_us":1658244}
I20250109 20:53:06.090574 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 225046d45c1c43b488babf5da8de75d7 (table default.Table_711789d44eb242309002efa5bd06ffb1 [id=a638ba7e8a3b4c309df4fd05ca948518]) successfully deleted
I20250109 20:53:06.092927 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:06.093143 1052 raft_consensus.cc:2463] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:06.091259 1165 raft_consensus.cc:357] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.093932 1165 raft_consensus.cc:383] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.094938 1165 raft_consensus.cc:738] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.096673 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e96fa48fa649909ef93a70a57c023c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.096997 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989e420afd094ff5adbe82ed4d75f7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.097182 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bfa04c33e104c148a73c545284c20a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.097404 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e54e9be016584fde911c4524969d0789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.097570 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452c685f82c94d8da488e15fac44d6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.097810 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7ca67649b64d26b9bf8341e6d59647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.098080 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87e5ff14595f4c9388d3d586ecfefbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.098346 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffba070fd36342369f4122da2b63eb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.100368 1165 consensus_queue.cc:260] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.109169 1165 ts_tablet_manager.cc:1428] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.021s user 0.002s sys 0.002s
I20250109 20:53:06.112423 1165 tablet_bootstrap.cc:492] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:06.120834 2032 raft_consensus.cc:491] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.120958 2032 raft_consensus.cc:513] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.121506 2032 leader_election.cc:290] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.121477 713 leader_election.cc:304] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
W20250109 20:53:06.122390 854 leader_election.cc:343] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: cd23012af5574f37944f9e26c07d05d2
I20250109 20:53:06.124397 1176 tablet_bootstrap.cc:654] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.127851 2413 raft_consensus.cc:2798] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:06.127949 2413 raft_consensus.cc:491] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:06.128013 2413 raft_consensus.cc:3054] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:06.129300 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1263) took 1976 ms. Trace:
I20250109 20:53:06.129390 1157 rpcz_store.cc:276] 0109 20:53:04.153091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.153136 (+ 45us) service_pool.cc:225] Handling call
0109 20:53:06.089774 (+1936638us) ts_tablet_manager.cc:1145] Deleting tablet cb880d4da1ca4c2c851fc8c1e0846ce4
0109 20:53:06.089786 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.129281 (+ 39495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1936495}
I20250109 20:53:06.129741 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1264) took 1967 ms. Trace:
I20250109 20:53:06.129812 1157 rpcz_store.cc:276] 0109 20:53:04.162153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.162197 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:06.129636 (+1967439us) ts_tablet_manager.cc:1145] Deleting tablet d79437d800f44c5a8222eea7dc2cd4e3
0109 20:53:06.129648 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.129729 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967273}
W20250109 20:53:06.130736 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb880d4da1ca4c2c851fc8c1e0846ce4 already in progress: creating tablet
W20250109 20:53:06.130988 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d79437d800f44c5a8222eea7dc2cd4e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d79437d800f44c5a8222eea7dc2cd4e3 already in progress: creating tablet
W20250109 20:53:06.132428 712 leader_election.cc:343] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.132638 712 leader_election.cc:304] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.134701 2533 raft_consensus.cc:2743] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:06.133903 713 leader_election.cc:343] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 724d15e05a864852a118badae13a5d3d
W20250109 20:53:06.138258 713 leader_election.cc:343] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8539cefebb844b3098a308ca580b0e92
W20250109 20:53:06.148059 712 leader_election.cc:343] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:06.148582 713 leader_election.cc:343] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 0cede2a1882946a181b18d66f12c5114
I20250109 20:53:06.153700 1165 tablet_bootstrap.cc:654] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.153667 1160 ts_tablet_manager.cc:1918] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:06.153961 1160 log.cc:1198] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/35a39fac67894a208dd7ec3153b05e4a
I20250109 20:53:06.154222 1160 ts_tablet_manager.cc:1939] T 35a39fac67894a208dd7ec3153b05e4a P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:06.155995 2475 raft_consensus.cc:491] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.156139 2475 raft_consensus.cc:513] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.157291 2475 leader_election.cc:290] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:06.157873 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "face19003d0042669e3f22553f9fe19a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:06.158543 989 leader_election.cc:343] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.149008 712 leader_election.cc:304] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.159942 1176 tablet_bootstrap.cc:492] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
W20250109 20:53:06.153867 713 leader_election.cc:343] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 5f953f43877c4a738ffd8255afbf8275
W20250109 20:53:06.160583 713 leader_election.cc:343] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 82209a6b39bb48e791ff63a8167c9765
I20250109 20:53:06.160915 1176 ts_tablet_manager.cc:1397] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.401s user 0.008s sys 0.030s
I20250109 20:53:06.161371 2488 raft_consensus.cc:2743] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.164603 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1261) took 2013 ms. Trace:
I20250109 20:53:06.165506 2534 raft_consensus.cc:491] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.165638 2534 raft_consensus.cc:513] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.164687 1157 rpcz_store.cc:276] 0109 20:53:04.150823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.176715 (+ 25892us) service_pool.cc:225] Handling call
0109 20:53:06.130030 (+1953315us) ts_tablet_manager.cc:1145] Deleting tablet 7357b4f67dc6448dbb8f77dce81731f3
0109 20:53:06.130039 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.164585 (+ 34546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953102}
W20250109 20:53:06.166786 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7357b4f67dc6448dbb8f77dce81731f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7357b4f67dc6448dbb8f77dce81731f3 already in progress: creating tablet
I20250109 20:53:06.168362 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "face19003d0042669e3f22553f9fe19a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:06.168856 990 leader_election.cc:343] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.168963 990 leader_election.cc:304] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.169209 2475 raft_consensus.cc:2743] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.176159 990 leader_election.cc:304] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.176442 2475 raft_consensus.cc:2798] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:06.176535 2475 raft_consensus.cc:491] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:06.176607 2475 raft_consensus.cc:3054] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:06.168181 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1267) took 1993 ms. Trace:
I20250109 20:53:06.179767 1157 rpcz_store.cc:276] 0109 20:53:04.174368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.174413 (+ 45us) service_pool.cc:225] Handling call
0109 20:53:06.166361 (+1991948us) ts_tablet_manager.cc:1145] Deleting tablet ac867ce748f749d88885ef13654983ec
0109 20:53:06.166374 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.168165 (+ 1791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1985900}
I20250109 20:53:06.180179 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1278) took 1862 ms. Trace:
W20250109 20:53:06.180276 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ac867ce748f749d88885ef13654983ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac867ce748f749d88885ef13654983ec already in progress: creating tablet
I20250109 20:53:06.180258 1157 rpcz_store.cc:276] 0109 20:53:04.318102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.318244 (+ 142us) service_pool.cc:225] Handling call
0109 20:53:06.180056 (+1861812us) ts_tablet_manager.cc:1145] Deleting tablet bd642e5ec83a4da88eaa2eea3ffca362
0109 20:53:06.180066 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.180163 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1861602}
W20250109 20:53:06.180891 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet bd642e5ec83a4da88eaa2eea3ffca362 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd642e5ec83a4da88eaa2eea3ffca362 already in progress: creating tablet
I20250109 20:53:06.182514 2534 leader_election.cc:290] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.183220 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1288) took 1106 ms. Trace:
I20250109 20:53:06.183365 1157 rpcz_store.cc:276] 0109 20:53:05.076990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.077035 (+ 45us) service_pool.cc:225] Handling call
0109 20:53:06.180466 (+1103431us) ts_tablet_manager.cc:1145] Deleting tablet 801838c21f70424ba1b2d89585bda643
0109 20:53:06.180474 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.183203 (+ 2729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103274}
W20250109 20:53:06.182535 712 leader_election.cc:343] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:06.183897 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 801838c21f70424ba1b2d89585bda643 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 801838c21f70424ba1b2d89585bda643 already in progress: creating tablet
I20250109 20:53:06.187575 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8ff04093442e413abe8433e8784b254f
I20250109 20:53:06.188423 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff04093442e413abe8433e8784b254f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.188652 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff04093442e413abe8433e8784b254f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.189625 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1289) took 1112 ms. Trace:
I20250109 20:53:06.189713 1157 rpcz_store.cc:276] 0109 20:53:05.077365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.077402 (+ 37us) service_pool.cc:225] Handling call
0109 20:53:06.183652 (+1106250us) ts_tablet_manager.cc:1145] Deleting tablet d0dc4addb7fd4d23822b25af5e288118
0109 20:53:06.183660 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.189602 (+ 5942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106125}
W20250109 20:53:06.190146 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d0dc4addb7fd4d23822b25af5e288118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0dc4addb7fd4d23822b25af5e288118 already in progress: creating tablet
I20250109 20:53:06.190372 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ff04093442e413abe8433e8784b254f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.190665 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1290) took 1113 ms. Trace:
I20250109 20:53:06.190723 1157 rpcz_store.cc:276] 0109 20:53:05.077679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.077716 (+ 37us) service_pool.cc:225] Handling call
0109 20:53:06.190557 (+1112841us) ts_tablet_manager.cc:1145] Deleting tablet 75ad55b91a6c405aae3ed456a8c4d9a3
0109 20:53:06.190569 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.190653 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112719}
W20250109 20:53:06.191020 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ad55b91a6c405aae3ed456a8c4d9a3 already in progress: creating tablet
I20250109 20:53:06.191711 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1291) took 1113 ms. Trace:
I20250109 20:53:06.191773 1157 rpcz_store.cc:276] 0109 20:53:05.077979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.078017 (+ 38us) service_pool.cc:225] Handling call
0109 20:53:06.191619 (+1113602us) ts_tablet_manager.cc:1145] Deleting tablet b087b7546a39451089666a36d029e645
0109 20:53:06.191627 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.191698 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113476}
W20250109 20:53:06.192063 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b087b7546a39451089666a36d029e645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b087b7546a39451089666a36d029e645 already in progress: creating tablet
W20250109 20:53:06.193267 712 leader_election.cc:343] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.193310 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1292) took 1115 ms. Trace:
I20250109 20:53:06.193358 712 leader_election.cc:304] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.193372 1157 rpcz_store.cc:276] 0109 20:53:05.078232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.078266 (+ 34us) service_pool.cc:225] Handling call
0109 20:53:06.193221 (+1114955us) ts_tablet_manager.cc:1145] Deleting tablet c6bc95b32aa04e25bbde02bf5e8fe4d9
0109 20:53:06.193229 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.193296 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1114826}
W20250109 20:53:06.193714 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 already in progress: creating tablet
I20250109 20:53:06.194584 2534 raft_consensus.cc:2743] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:06.195916 712 leader_election.cc:343] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.196005 712 leader_election.cc:304] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:06.196503 713 leader_election.cc:343] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 28cd9cd48910497ba3e4b159583cb7a6
I20250109 20:53:06.198400 2534 raft_consensus.cc:2743] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:06.199043 712 leader_election.cc:343] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.199121 712 leader_election.cc:304] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.199276 2534 raft_consensus.cc:2743] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.200323 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 49e0863249364194936804ae83645936
I20250109 20:53:06.201146 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_029561426fdb42bba6f6ec539f655c01" } new_table_name: "default.Table_2dfa13fd188a4d5481559436ffe57a0a" modify_external_catalogs: true
I20250109 20:53:06.202105 1157 tablet_replica.cc:331] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:06.202491 1157 raft_consensus.cc:2238] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:06.202684 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88a31e2d5bc94f82ae1cbdf6cac5d77c consensus state has no leader
I20250109 20:53:06.202689 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e0863249364194936804ae83645936 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.202962 1157 raft_consensus.cc:2267] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:06.203029 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1f0817571af249be925827a9c80c6ffd" } new_table_name: "default.Table_0cb4533c806c4cc08135f91aa53ef1a4" modify_external_catalogs: true
I20250109 20:53:06.203527 1157 ts_tablet_manager.cc:1905] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:06.205685 1158 tablet_bootstrap.cc:492] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:06.205813 1158 ts_tablet_manager.cc:1397] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.189s user 0.001s sys 0.005s
I20250109 20:53:06.206492 1158 raft_consensus.cc:357] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.206692 1158 raft_consensus.cc:383] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.206743 1158 raft_consensus.cc:738] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.206924 1158 consensus_queue.cc:260] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.207386 1158 ts_tablet_manager.cc:1428] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 20:53:06.207617 1158 tablet_bootstrap.cc:492] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:06.208186 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7a4b7e008a474978b91426f63dbd96ee
I20250109 20:53:06.209163 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_181b0576d1844f09b0045ac8d5f16841"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.209743 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_181b0576d1844f09b0045ac8d5f16841 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.210772 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4b7e008a474978b91426f63dbd96ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.212080 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da1f3cafffe4f7ca39624b43831689e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.212251 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b339612e60ae46b48f88dc6d43a1a530" } modify_external_catalogs: true
I20250109 20:53:06.213956 1158 tablet_bootstrap.cc:654] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:06.214787 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e680808eea0416bb05d5d61e0cc055d consensus state has no leader
I20250109 20:53:06.215553 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fe13a1782fc74940871718d79b25d8b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.216128 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe13a1782fc74940871718d79b25d8b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:06.221050 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6b23623677b4e088c14af925b4e07d4 consensus state has no leader
W20250109 20:53:06.230737 854 leader_election.cc:343] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: daac99da5d10475c9de263bef27baeb7
I20250109 20:53:06.230867 854 leader_election.cc:304] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:06.231274 854 leader_election.cc:343] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 728e9b6101e3447fb3c24b3917aa4872
I20250109 20:53:06.231367 854 leader_election.cc:304] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.231384 919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82209a6b39bb48e791ff63a8167c9765" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:06.231643 2032 raft_consensus.cc:2743] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.231792 2032 raft_consensus.cc:2743] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.233106 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28cd9cd48910497ba3e4b159583cb7a6" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:06.233613 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f953f43877c4a738ffd8255afbf8275" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:06.234539 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e0863249364194936804ae83645936 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.235468 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4b7e008a474978b91426f63dbd96ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.238085 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e0863249364194936804ae83645936 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.238520 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4b7e008a474978b91426f63dbd96ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.243830 1176 raft_consensus.cc:357] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.244899 1176 raft_consensus.cc:383] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.245147 1176 raft_consensus.cc:738] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.245522 1176 consensus_queue.cc:260] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.247241 1176 ts_tablet_manager.cc:1428] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.086s user 0.002s sys 0.002s
W20250109 20:53:06.247478 712 leader_election.cc:343] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.247627 712 leader_election.cc:304] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.247699 2413 raft_consensus.cc:513] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.247906 2534 raft_consensus.cc:2743] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.248276 2413 leader_election.cc:290] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.249271 1176 tablet_bootstrap.cc:492] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:53:06.254212 854 leader_election.cc:343] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 57e56f30e6f3450a8426cd40db89e009
I20250109 20:53:06.254541 854 leader_election.cc:304] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:06.254874 854 leader_election.cc:343] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: cd23012af5574f37944f9e26c07d05d2
I20250109 20:53:06.254949 854 leader_election.cc:304] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:06.255116 712 leader_election.cc:343] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.255149 2032 raft_consensus.cc:2743] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.255189 1165 tablet_bootstrap.cc:492] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.255204 712 leader_election.cc:304] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.255283 2032 raft_consensus.cc:2743] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.255302 1165 ts_tablet_manager.cc:1397] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.144s user 0.008s sys 0.021s
I20250109 20:53:06.255378 2534 raft_consensus.cc:2743] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:06.255637 854 leader_election.cc:343] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: ca816507d0c047668a51f6e7ebce18cb
I20250109 20:53:06.255723 854 leader_election.cc:304] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.255936 2032 raft_consensus.cc:2743] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:06.256042 712 leader_election.cc:343] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.256129 712 leader_election.cc:304] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.256070 1165 raft_consensus.cc:357] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.256240 1165 raft_consensus.cc:383] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.256286 1165 raft_consensus.cc:738] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.256301 2534 raft_consensus.cc:2743] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.256464 1165 consensus_queue.cc:260] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.247399 2475 raft_consensus.cc:513] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:06.258157 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 57994.5us
I20250109 20:53:06.258481 2475 leader_election.cc:290] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:06.259059 1158 tablet_bootstrap.cc:492] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:06.259162 1158 ts_tablet_manager.cc:1397] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.005s
I20250109 20:53:06.259301 1176 tablet_bootstrap.cc:654] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.259732 1158 raft_consensus.cc:357] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.259950 1158 raft_consensus.cc:383] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.260003 1158 raft_consensus.cc:738] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.260318 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:06.260463 1052 raft_consensus.cc:3054] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:06.261232 1165 ts_tablet_manager.cc:1428] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:53:06.263635 1165 tablet_bootstrap.cc:492] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:06.265439 1158 consensus_queue.cc:260] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.265822 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3c276fab6a4d068d64ea5d23c9a595" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:06.266091 1158 ts_tablet_manager.cc:1428] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250109 20:53:06.268186 989 leader_election.cc:343] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.268476 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:06.270324 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1162) took 2768 ms. Trace:
I20250109 20:53:06.270402 1160 rpcz_store.cc:276] 0109 20:53:03.502361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.502405 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:05.432814 (+1930409us) ts_tablet_manager.cc:1145] Deleting tablet 35a39fac67894a208dd7ec3153b05e4a
0109 20:53:05.432823 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:05.934300 (+501477us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.153575 (+219275us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.270309 (+116734us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":203789,"tablet-delete.queue_time_us":1930251}
I20250109 20:53:06.270797 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 35a39fac67894a208dd7ec3153b05e4a (table default.Table_df4e58889f9d4ef6b81876c48ecf223e [id=fcb76535fbbe44e29ef627a3ae562ce2]) successfully deleted
I20250109 20:53:06.271396 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1176) took 2595 ms. Trace:
I20250109 20:53:06.271798 1160 rpcz_store.cc:276] 0109 20:53:03.676194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.679908 (+ 3714us) service_pool.cc:225] Handling call
0109 20:53:06.271293 (+2591385us) ts_tablet_manager.cc:1145] Deleting tablet 9c3af2a00ab04b41915cffc3eb9206bc
0109 20:53:06.271302 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.271381 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2591150}
W20250109 20:53:06.272192 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9c3af2a00ab04b41915cffc3eb9206bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c3af2a00ab04b41915cffc3eb9206bc already in progress: creating tablet
I20250109 20:53:06.272388 2368 raft_consensus.cc:491] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.272526 2368 raft_consensus.cc:513] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.273358 2368 leader_election.cc:290] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:06.273957 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1177) took 2597 ms. Trace:
I20250109 20:53:06.274001 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcb6780422541c89343e98e02938d0e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:06.274024 1160 rpcz_store.cc:276] 0109 20:53:03.676296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.680175 (+ 3879us) service_pool.cc:225] Handling call
0109 20:53:06.273860 (+2593685us) ts_tablet_manager.cc:1145] Deleting tablet d2c8a6555a8e4228a9a80b15c6880c84
0109 20:53:06.273869 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.273942 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2593519}
W20250109 20:53:06.274215 712 leader_election.cc:343] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:06.274530 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d2c8a6555a8e4228a9a80b15c6880c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c8a6555a8e4228a9a80b15c6880c84 already in progress: creating tablet
I20250109 20:53:06.274860 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1173) took 2599 ms. Trace:
I20250109 20:53:06.274925 1160 rpcz_store.cc:276] 0109 20:53:03.675892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.680365 (+ 4473us) service_pool.cc:225] Handling call
0109 20:53:06.274753 (+2594388us) ts_tablet_manager.cc:1145] Deleting tablet 5f953f43877c4a738ffd8255afbf8275
0109 20:53:06.274762 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.274845 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2594231}
W20250109 20:53:06.275255 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5f953f43877c4a738ffd8255afbf8275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f953f43877c4a738ffd8255afbf8275 already in progress: creating tablet
I20250109 20:53:06.275583 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1174) took 2599 ms. Trace:
I20250109 20:53:06.275648 1160 rpcz_store.cc:276] 0109 20:53:03.676004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.683636 (+ 7632us) service_pool.cc:225] Handling call
0109 20:53:06.275473 (+2591837us) ts_tablet_manager.cc:1145] Deleting tablet 3293f49190684b72a5c41031e621bd36
0109 20:53:06.275480 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.275566 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2590876}
W20250109 20:53:06.276033 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3293f49190684b72a5c41031e621bd36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3293f49190684b72a5c41031e621bd36 already in progress: creating tablet
I20250109 20:53:06.276610 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1175) took 2600 ms. Trace:
I20250109 20:53:06.276763 1160 rpcz_store.cc:276] 0109 20:53:03.676109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.684449 (+ 8340us) service_pool.cc:225] Handling call
0109 20:53:06.276512 (+2592063us) ts_tablet_manager.cc:1145] Deleting tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6
0109 20:53:06.276522 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.276594 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2591706}
W20250109 20:53:06.277209 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 already in progress: creating tablet
I20250109 20:53:06.277491 1160 tablet_replica.cc:331] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:06.277688 1160 raft_consensus.cc:2238] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:06.277931 1160 raft_consensus.cc:2267] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:06.278147 2534 raft_consensus.cc:491] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.278262 2534 raft_consensus.cc:513] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.278851 2534 leader_election.cc:290] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.279448 2223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=785 overwritten=0 applied=785 ignored=0} inserts{seen=743 ignored=0} mutations{seen=895 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250109 20:53:06.279824 1160 ts_tablet_manager.cc:1905] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:06.281643 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3c276fab6a4d068d64ea5d23c9a595" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:06.281666 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcb6780422541c89343e98e02938d0e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:06.281821 778 raft_consensus.cc:3054] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:06.282124 990 leader_election.cc:343] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.282212 1158 tablet_bootstrap.cc:492] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:06.282449 1176 tablet_bootstrap.cc:492] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.282618 1176 ts_tablet_manager.cc:1397] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.005s
I20250109 20:53:06.282806 2223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: Bootstrap complete.
I20250109 20:53:06.283825 1158 tablet_bootstrap.cc:654] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.283602 1176 raft_consensus.cc:357] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.284142 1176 raft_consensus.cc:383] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.284224 1176 raft_consensus.cc:738] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.284528 1176 consensus_queue.cc:260] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.285545 1176 ts_tablet_manager.cc:1428] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:53:06.286422 1176 tablet_bootstrap.cc:492] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.289131 1176 tablet_bootstrap.cc:654] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.291915 1158 tablet_bootstrap.cc:492] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:06.281081 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec1d30b67674facb8b92a2c1a408073" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:06.292403 712 leader_election.cc:343] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.292884 1158 ts_tablet_manager.cc:1397] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20250109 20:53:06.305430 823 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:06.293789 1158 raft_consensus.cc:357] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.306005 1158 raft_consensus.cc:383] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.306272 1158 raft_consensus.cc:738] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.306617 1158 consensus_queue.cc:260] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.307919 1158 ts_tablet_manager.cc:1428] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250109 20:53:06.307919 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_40f47b080f4a4134ad0921f8bcfe2f23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.308493 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40f47b080f4a4134ad0921f8bcfe2f23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:06.309296 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6880a3e914414c77bf7682da4057df4c consensus state has no leader
I20250109 20:53:06.308409 2556 tablet.cc:1721] T 1c65489a602141239c99e8e8dae22ab4 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:06.311599 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 1c65489a602141239c99e8e8dae22ab4 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]). Expected version 4 got 0
I20250109 20:53:06.312386 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_494f8c109277462ab1dc23afe0455989"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.312968 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_494f8c109277462ab1dc23afe0455989 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.312999 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ffcefcb7b4a548e085267ae98346f908"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.313504 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffcefcb7b4a548e085267ae98346f908 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.314071 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_27dce06a871149eb94e251f6e60e048d" } new_table_name: "default.Table_dadeccdf43324b11a72aad7f2d19b825" modify_external_catalogs: true
I20250109 20:53:06.314682 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8a0b2eddb0384893be34d4820890256c" } modify_external_catalogs: true
I20250109 20:53:06.314800 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8a0b2eddb0384893be34d4820890256c" } modify_external_catalogs: true
W20250109 20:53:06.319831 989 leader_election.cc:343] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.322903 1165 tablet_bootstrap.cc:654] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.325147 989 leader_election.cc:304] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.332862 2368 raft_consensus.cc:2743] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:06.333886 713 leader_election.cc:343] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 0ec1d30b67674facb8b92a2c1a408073
I20250109 20:53:06.334112 713 leader_election.cc:304] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.334496 2534 raft_consensus.cc:2743] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.334985 2553 tablet.cc:1721] T 7b7dc6809d6547f6b0354b64d7149bf4 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:06.341604 2223 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:53:06.342651 2223 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a29ab2002d4eb389cd11778988e1e9, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.342661 1052 raft_consensus.cc:2463] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:06.343366 2223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 785, Last appended: 1.785, Last appended by leader: 785, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:53:06.342816 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_247c9acc11f648b281616ab9d9ea7fc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.344170 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_247c9acc11f648b281616ab9d9ea7fc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.347906 778 raft_consensus.cc:2463] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:06.348065 2560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:53:06.348351 2560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:53:06.348400 990 leader_election.cc:304] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.348901 2368 raft_consensus.cc:2798] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:06.349012 2368 raft_consensus.cc:695] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:53:06.349237 2368 consensus_queue.cc:237] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.350236 2223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: configured and running, proceeding with master startup.
I20250109 20:53:06.350826 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 0538f4d3608a454b9dd21a0fa42068a0 (table default.Table_09ed969c8f9a442392025cc69ce2ea70 [id=3fde04a3af42412db45f2c91950fb8e4]). Expected version 2 got 0
I20250109 20:53:06.365428 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b50c91807734e86840e874886dc0749 with delete_type TABLET_DATA_DELETED (replaced by tablet 39f0cc2be25940d08fad42a8c8d8a07b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.365942 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 3293f49190684b72a5c41031e621bd36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.376643 713 leader_election.cc:304] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.379540 2413 raft_consensus.cc:2798] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:06.379724 2413 raft_consensus.cc:695] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:06.339200 1165 tablet_bootstrap.cc:492] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.347007 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e46f4312844b6eb29d6718f4800f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.381139 1165 ts_tablet_manager.cc:1397] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.118s user 0.002s sys 0.016s
I20250109 20:53:06.380005 2413 consensus_queue.cc:237] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.381917 1165 raft_consensus.cc:357] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.382092 1165 raft_consensus.cc:383] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.382159 1165 raft_consensus.cc:738] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.382351 1165 consensus_queue.cc:260] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.382807 1165 ts_tablet_manager.cc:1428] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:06.383126 1165 tablet_bootstrap.cc:492] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:06.384366 1165 tablet_bootstrap.cc:654] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.384526 1176 tablet_bootstrap.cc:492] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.384646 1176 ts_tablet_manager.cc:1397] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.015s
I20250109 20:53:06.385295 1176 raft_consensus.cc:357] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.385478 1176 raft_consensus.cc:383] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.385525 1176 raft_consensus.cc:738] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.385721 1176 consensus_queue.cc:260] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.386184 1176 ts_tablet_manager.cc:1428] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:53:06.386423 1176 tablet_bootstrap.cc:492] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.387655 1176 tablet_bootstrap.cc:654] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.365101 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 190ccaff0ca7486f9dac755f2f625677
I20250109 20:53:06.392407 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 190ccaff0ca7486f9dac755f2f625677 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.393151 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 190ccaff0ca7486f9dac755f2f625677 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:06.365283 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b6b23623677b4e088c14af925b4e07d4 (table default.Table_2dfa13fd188a4d5481559436ffe57a0a [id=3ae741f5fe4343188b6f0d10b2a09b69], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b6b23623677b4e088c14af925b4e07d4 consensus state has no leader
I20250109 20:53:06.396934 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 190ccaff0ca7486f9dac755f2f625677 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.396934 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 8890a6a93e3d48dc96955f1b51e78603 with delete_type TABLET_DATA_DELETED (replaced by tablet beca5f1ed9d3434fb7de90cd74a6f09b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.397801 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8a70d90c9fb04010bfc9fb5342c82862" } modify_external_catalogs: true
I20250109 20:53:06.397956 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8a70d90c9fb04010bfc9fb5342c82862" } modify_external_catalogs: true
I20250109 20:53:06.398449 1176 tablet_bootstrap.cc:492] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.398552 1176 ts_tablet_manager.cc:1397] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250109 20:53:06.375841 449 catalog_manager.cc:5526] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
W20250109 20:53:06.377619 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 156fbcdb4c9b4f83a96dad732a9a0964 consensus state has no leader
I20250109 20:53:06.399102 1176 raft_consensus.cc:357] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.399255 1176 raft_consensus.cc:383] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.399312 1176 raft_consensus.cc:738] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.399430 449 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (table default.Table_159068abc4d842c693e5ae2323d1147d [id=15c9f7d5d3ff4fa4865b55c5892421fb]). Expected version 3 got 0
I20250109 20:53:06.378751 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 204ae923b8354fc791b5c4fb7197dc76
I20250109 20:53:06.399521 1176 consensus_queue.cc:260] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.399643 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 1c65489a602141239c99e8e8dae22ab4 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]). Expected version 4 got 0
I20250109 20:53:06.399960 1176 ts_tablet_manager.cc:1428] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:06.400054 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ae923b8354fc791b5c4fb7197dc76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.400184 1176 tablet_bootstrap.cc:492] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.400563 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ae923b8354fc791b5c4fb7197dc76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.400966 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ae923b8354fc791b5c4fb7197dc76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.401728 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:53:06.401964 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bc7963618a2e45d6999e681eec0db896" } modify_external_catalogs: true
I20250109 20:53:06.402037 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]). Expected version 1 got 0
I20250109 20:53:06.402107 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bc7963618a2e45d6999e681eec0db896" } modify_external_catalogs: true
I20250109 20:53:06.402387 1176 tablet_bootstrap.cc:654] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.405946 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_385822fe7219471bacdd1ad2dbe7d891" } new_table_name: "default.Table_f0324cc5041947eea59343a255ed227b" modify_external_catalogs: true
I20250109 20:53:06.407351 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.410650 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a2b81fc56f9c469e9b72ed20afcb7e02" } modify_external_catalogs: true
I20250109 20:53:06.410784 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a2b81fc56f9c469e9b72ed20afcb7e02" } modify_external_catalogs: true
I20250109 20:53:06.415588 2534 consensus_queue.cc:1035] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:06.416060 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_62c8e60a5c084b149750c3b8acb15aa8" } new_table_name: "default.Table_116046375f9249cd9761f3719b5967e4" modify_external_catalogs: true
I20250109 20:53:06.417529 1158 tablet_bootstrap.cc:492] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:06.419437 1158 tablet_bootstrap.cc:654] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.422061 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2e425a9a8d0f49acab65342112e3e64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:06.422258 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a017309fe2384e848ad25b4288eb890b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.422536 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e425a9a8d0f49acab65342112e3e64c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.422734 1176 tablet_bootstrap.cc:492] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
W20250109 20:53:06.422710 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a017309fe2384e848ad25b4288eb890b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.422840 1176 ts_tablet_manager.cc:1397] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250109 20:53:06.423445 1176 raft_consensus.cc:357] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.423607 1176 raft_consensus.cc:383] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.423655 1176 raft_consensus.cc:738] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.424212 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_33baa454083e4b7eb40552ba4fc70124"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.424607 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33baa454083e4b7eb40552ba4fc70124 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.425835 2534 raft_consensus.cc:491] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.425966 2534 raft_consensus.cc:513] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.426234 1176 consensus_queue.cc:260] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.426923 1176 ts_tablet_manager.cc:1428] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:53:06.427156 1176 tablet_bootstrap.cc:492] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.427107 2534 leader_election.cc:290] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.428278 1176 tablet_bootstrap.cc:654] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.429311 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 64d462969643467c97222cac8adc210d
I20250109 20:53:06.430596 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d462969643467c97222cac8adc210d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.431174 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ea166542339a4980aea4afc8f128edd5" } new_table_name: "default.Table_aaeeff82c025462498c8e9d4b583426b" modify_external_catalogs: true
I20250109 20:53:06.432720 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d462969643467c97222cac8adc210d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.432996 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d462969643467c97222cac8adc210d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:06.444788 712 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.0.104.60:41583, user_credentials={real_user=slave}} blocked reactor thread for 139386us
I20250109 20:53:06.446686 2469 raft_consensus.cc:491] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.446743 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74a4c1bdd8f4f6e9592ee5d97c58cc6" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:06.446821 2469 raft_consensus.cc:513] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.447387 2469 leader_election.cc:290] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:06.448285 712 leader_election.cc:343] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.448446 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57a8bbb3da84088a98c816102e4e9df" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:06.449782 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 795f0f29f0744f4e9dd2fcf1762ab978 consensus state has no leader
I20250109 20:53:06.452212 1166 ts_tablet_manager.cc:1918] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:06.452360 1166 log.cc:1198] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/404ee892ab79427baaef6d19c57699d0
I20250109 20:53:06.452702 1166 ts_tablet_manager.cc:1939] T 404ee892ab79427baaef6d19c57699d0 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:06.453009 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1222) took 2558 ms. Trace:
I20250109 20:53:06.453074 1166 rpcz_store.cc:276] 0109 20:53:03.894913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.895552 (+ 639us) service_pool.cc:225] Handling call
0109 20:53:05.739933 (+1844381us) ts_tablet_manager.cc:1145] Deleting tablet 404ee892ab79427baaef6d19c57699d0
0109 20:53:05.739943 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.357263 (+617320us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.452063 (+ 94800us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.452990 (+ 927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1844165}
I20250109 20:53:06.453514 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 404ee892ab79427baaef6d19c57699d0 (table default.Table_3abc7e662927447b90d40c4c4ba6e04b [id=d394131690ff4ebaa3d9639758b62f04]) successfully deleted
I20250109 20:53:06.454007 1166 tablet_replica.cc:331] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:06.454147 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6492caf3355c4685a6f0670ad1a352e9
I20250109 20:53:06.454192 1166 raft_consensus.cc:2238] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:06.454344 1166 raft_consensus.cc:2267] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:06.455307 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 6492caf3355c4685a6f0670ad1a352e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.455986 1166 ts_tablet_manager.cc:1905] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:06.458456 1157 ts_tablet_manager.cc:1918] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:06.458475 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6492caf3355c4685a6f0670ad1a352e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.458753 1157 log.cc:1198] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/8539cefebb844b3098a308ca580b0e92
I20250109 20:53:06.459084 1157 ts_tablet_manager.cc:1939] T 8539cefebb844b3098a308ca580b0e92 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:06.460137 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 6492caf3355c4685a6f0670ad1a352e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.464346 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.466306 1176 tablet_bootstrap.cc:492] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.466428 1176 ts_tablet_manager.cc:1397] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20250109 20:53:06.468109 2368 consensus_queue.cc:1035] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:53:06.468286 712 leader_election.cc:343] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:06.468451 712 consensus_peers.cc:487] T 8e6a9aba1c7b424db21c30b2e92323c9 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:06.469908 1176 raft_consensus.cc:357] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.470109 1176 raft_consensus.cc:383] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.470160 1176 raft_consensus.cc:738] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.470319 1176 consensus_queue.cc:260] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.475044 1176 ts_tablet_manager.cc:1428] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:53:06.475292 1176 tablet_bootstrap.cc:492] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.476670 1176 tablet_bootstrap.cc:654] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:06.489578 713 leader_election.cc:343] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: b74a4c1bdd8f4f6e9592ee5d97c58cc6
I20250109 20:53:06.489878 713 leader_election.cc:304] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.490031 1158 tablet_bootstrap.cc:492] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:06.490279 1158 ts_tablet_manager.cc:1397] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.002s
W20250109 20:53:06.490568 713 leader_election.cc:343] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: d57a8bbb3da84088a98c816102e4e9df
I20250109 20:53:06.490690 713 leader_election.cc:304] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.490916 2533 raft_consensus.cc:2743] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.491081 2533 raft_consensus.cc:2743] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.491137 1158 raft_consensus.cc:357] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.491307 1158 raft_consensus.cc:383] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.491354 1158 raft_consensus.cc:738] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.491516 1158 consensus_queue.cc:260] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.491961 1158 ts_tablet_manager.cc:1428] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:06.492206 1158 tablet_bootstrap.cc:492] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:06.495127 2558 tablet.cc:1721] T 1c65489a602141239c99e8e8dae22ab4 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:06.498911 1158 tablet_bootstrap.cc:654] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:06.498663 989 consensus_peers.cc:487] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250109 20:53:06.500903 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a9912ba97e04666bba5911cd3d5c296 consensus state has no leader
I20250109 20:53:06.503005 1176 tablet_bootstrap.cc:492] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.503132 1176 ts_tablet_manager.cc:1397] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20250109 20:53:06.503906 1176 raft_consensus.cc:357] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.504096 1176 raft_consensus.cc:383] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.504158 1176 raft_consensus.cc:738] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.504364 1176 consensus_queue.cc:260] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.504566 2546 raft_consensus.cc:491] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.504662 2546 raft_consensus.cc:513] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.504884 1176 ts_tablet_manager.cc:1428] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:06.505111 1176 tablet_bootstrap.cc:492] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.505259 2546 leader_election.cc:290] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:06.505956 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c83e709a8d4e078e67af58ec51beac" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:06.506601 990 leader_election.cc:343] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.507385 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c83e709a8d4e078e67af58ec51beac" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:06.508055 989 leader_election.cc:343] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.508266 989 leader_election.cc:304] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.513511 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2dc990ff61804866a5a8ea9d127694e9" } new_table_name: "default.Table_415a06752c304e22b44849f3c7ac7fe1" modify_external_catalogs: true
I20250109 20:53:06.523334 2546 raft_consensus.cc:2743] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.524055 1165 tablet_bootstrap.cc:492] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.524195 1165 ts_tablet_manager.cc:1397] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.036s
I20250109 20:53:06.524806 1165 raft_consensus.cc:357] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.524988 1165 raft_consensus.cc:383] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.525031 1165 raft_consensus.cc:738] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.525204 1165 consensus_queue.cc:260] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.525656 1165 ts_tablet_manager.cc:1428] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:06.525910 1165 tablet_bootstrap.cc:492] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:06.526510 1176 tablet_bootstrap.cc:654] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.541661 1176 tablet_bootstrap.cc:492] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.541801 1176 ts_tablet_manager.cc:1397] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20250109 20:53:06.542477 1176 raft_consensus.cc:357] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.542675 1176 raft_consensus.cc:383] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.542729 1176 raft_consensus.cc:738] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.542953 1176 consensus_queue.cc:260] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.543469 1176 ts_tablet_manager.cc:1428] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:06.543596 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1293) took 1465 ms. Trace:
I20250109 20:53:06.543725 1176 tablet_bootstrap.cc:492] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.543723 1157 rpcz_store.cc:276] 0109 20:53:05.078489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.078521 (+ 32us) service_pool.cc:225] Handling call
0109 20:53:06.202045 (+1123524us) ts_tablet_manager.cc:1145] Deleting tablet 8539cefebb844b3098a308ca580b0e92
0109 20:53:06.202056 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.293534 (+ 91478us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.458350 (+164816us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.543571 (+ 85221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123411}
I20250109 20:53:06.546046 1165 tablet_bootstrap.cc:654] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.548444 1165 tablet_bootstrap.cc:492] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.548547 1165 ts_tablet_manager.cc:1397] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250109 20:53:06.548874 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 8539cefebb844b3098a308ca580b0e92 (table default.Table_1d52e3d34dc14e729a515dc24e0787d4 [id=b74c21f2cb874f2485c0e3bc1d6e0dc8]) successfully deleted
I20250109 20:53:06.549266 1165 raft_consensus.cc:357] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.549444 1165 raft_consensus.cc:383] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.549494 1165 raft_consensus.cc:738] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.549393 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1308) took 1456 ms. Trace:
I20250109 20:53:06.549666 1157 rpcz_store.cc:276] 0109 20:53:05.280239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.280289 (+ 50us) service_pool.cc:225] Handling call
0109 20:53:06.549260 (+1268971us) ts_tablet_manager.cc:1145] Deleting tablet 568e4a83fddd45c8b61849a369074c4d
0109 20:53:06.549271 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.549375 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268777}
I20250109 20:53:06.549703 1165 consensus_queue.cc:260] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.550194 1165 ts_tablet_manager.cc:1428] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250109 20:53:06.550196 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 568e4a83fddd45c8b61849a369074c4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 568e4a83fddd45c8b61849a369074c4d already in progress: creating tablet
I20250109 20:53:06.550438 1165 tablet_bootstrap.cc:492] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:06.550653 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1312) took 1269 ms. Trace:
I20250109 20:53:06.550773 1157 rpcz_store.cc:276] 0109 20:53:05.281458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.281498 (+ 40us) service_pool.cc:225] Handling call
0109 20:53:06.550455 (+1268957us) ts_tablet_manager.cc:1145] Deleting tablet aba42482feab4d0f85fb46b4c8b01d19
0109 20:53:06.550464 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.550637 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268816}
W20250109 20:53:06.551185 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet aba42482feab4d0f85fb46b4c8b01d19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aba42482feab4d0f85fb46b4c8b01d19 already in progress: creating tablet
I20250109 20:53:06.551651 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1320) took 1268 ms. Trace:
I20250109 20:53:06.551757 1157 rpcz_store.cc:276] 0109 20:53:05.282949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.282990 (+ 41us) service_pool.cc:225] Handling call
0109 20:53:06.551545 (+1268555us) ts_tablet_manager.cc:1145] Deleting tablet ec1eaa6bc5a84b69840c540319265efb
0109 20:53:06.551555 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.551638 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268409}
W20250109 20:53:06.552242 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ec1eaa6bc5a84b69840c540319265efb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec1eaa6bc5a84b69840c540319265efb already in progress: creating tablet
I20250109 20:53:06.552707 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1321) took 1269 ms. Trace:
I20250109 20:53:06.552840 1157 rpcz_store.cc:276] 0109 20:53:05.283240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.283277 (+ 37us) service_pool.cc:225] Handling call
0109 20:53:06.552604 (+1269327us) ts_tablet_manager.cc:1145] Deleting tablet 5f05e17ca65243a898a7c75d3f453706
0109 20:53:06.552614 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.552691 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269201}
W20250109 20:53:06.553277 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5f05e17ca65243a898a7c75d3f453706 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f05e17ca65243a898a7c75d3f453706 already in progress: creating tablet
I20250109 20:53:06.553642 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1322) took 1270 ms. Trace:
I20250109 20:53:06.553784 1157 rpcz_store.cc:276] 0109 20:53:05.283513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.283548 (+ 35us) service_pool.cc:225] Handling call
0109 20:53:06.553540 (+1269992us) ts_tablet_manager.cc:1145] Deleting tablet 908781d94a054386b7d7c06c2c9fa3ca
0109 20:53:06.553550 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.553626 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269850}
I20250109 20:53:06.557641 1176 tablet_bootstrap.cc:654] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:06.561950 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcba62cbadf04761822bf7161c52b758 consensus state has no leader
I20250109 20:53:06.562423 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c2360ad67a3340189132339b30692659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.562918 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2360ad67a3340189132339b30692659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.564087 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d0aaf004b27e42c8903f69a38b65289f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.564644 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0aaf004b27e42c8903f69a38b65289f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.565677 1165 tablet_bootstrap.cc:654] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.569164 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4f047e8fa34ca79e78915c61a8ba12 with delete_type TABLET_DATA_DELETED (replaced by tablet b6b23623677b4e088c14af925b4e07d4) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.569780 881 tablet_service.cc:1514] Processing DeleteTablet for tablet a0d09daa2eed41c3952fa21aa2cdd09e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.572175 447 catalog_manager.cc:5526] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:06.572494 447 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:06.572774 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet c0860554cac64b8882e39fa62462b1aa (table default.Table_f74ab9f194134e6ba390983fba09d20a [id=f36b9d8c432d4f108296cd5576832573]). Expected version 2 got 0
I20250109 20:53:06.574038 881 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3d64693f594ea7b367f5add0068aaf (DEFAULT_TABLE table=default.Table_3eb37633d0cd466a9e79e8fa50d54f03 [id=bbcf1ce2703d435e8bd783f22bf71683]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.574275 1176 tablet_bootstrap.cc:492] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.574394 1176 ts_tablet_manager.cc:1397] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250109 20:53:06.574985 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797e1df3d12476f89fb5155523c3935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.575106 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af9563f6efd47059caac7c86c41d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.575135 1176 raft_consensus.cc:357] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.575309 1176 raft_consensus.cc:383] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.575357 1176 raft_consensus.cc:738] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.575585 1176 consensus_queue.cc:260] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.576097 1176 ts_tablet_manager.cc:1428] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:06.576339 1176 tablet_bootstrap.cc:492] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.577344 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f77f35085ff43a5b7fb6cce7e7a3409 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:06.578426 1176 tablet_bootstrap.cc:654] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.578478 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a6bfa7b3b5654fe4ad9b3401476d068d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:06.578874 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 64427614c0ac40afb54343277cc578bb (DEFAULT_TABLE table=default.Table_a4bd298f3639422487e396ef57dd2756 [id=4d6c8c4ea93c4b7abd749a013462ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:06.578933 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6bfa7b3b5654fe4ad9b3401476d068d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.578903 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3d64693f594ea7b367f5add0068aaf (DEFAULT_TABLE table=default.Table_3eb37633d0cd466a9e79e8fa50d54f03 [id=bbcf1ce2703d435e8bd783f22bf71683]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.579366 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e9289eacd1074dd59e244001042ccbe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.579692 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9289eacd1074dd59e244001042ccbe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.579949 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_31fe662c8ac540c69f62079a1cf3a9e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.580341 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31fe662c8ac540c69f62079a1cf3a9e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.581938 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4a39059d6b274f35a4a076611c7fb828 (DEFAULT_TABLE table=default.Table_591e8768478a4bd28f197e2b94197c39 [id=7ea97514ee88484dbf97f4a0ec6e584d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.581939 887 tablet_service.cc:1467] Processing CreateTablet for tablet 64427614c0ac40afb54343277cc578bb (DEFAULT_TABLE table=default.Table_a4bd298f3639422487e396ef57dd2756 [id=4d6c8c4ea93c4b7abd749a013462ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.582901 2552 raft_consensus.cc:491] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.583007 2552 raft_consensus.cc:513] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.583065 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 4a39059d6b274f35a4a076611c7fb828 (DEFAULT_TABLE table=default.Table_591e8768478a4bd28f197e2b94197c39 [id=7ea97514ee88484dbf97f4a0ec6e584d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.583536 2552 leader_election.cc:290] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.584422 886 tablet_service.cc:1467] Processing CreateTablet for tablet d029b5e24a91485a989ff462a678c4a2 (DEFAULT_TABLE table=default.Table_21a5abc833a749e487f214e77ffb3938 [id=eac56f9087164d8c81c85cf9996724b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:06.584645 854 leader_election.cc:343] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 7da1f3cafffe4f7ca39624b43831689e
W20250109 20:53:06.585819 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70f06752b26c47f49d8486adc6c4d92c consensus state has no leader
I20250109 20:53:06.587342 883 tablet_service.cc:1467] Processing CreateTablet for tablet b924c0bdfda24e3ca0a78e60a925bcd1 (DEFAULT_TABLE table=default.Table_6da66d9b734d4dfc848a5ae9e5e184dd [id=e31b1a30b94f4f7da3ff4b6112445435]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.589900 888 tablet_service.cc:1467] Processing CreateTablet for tablet a2fb9168f0c74525807bc72b888cb35a (DEFAULT_TABLE table=default.Table_db4cd004494744f892fc049ed4e65a4e [id=18a9e0fa4a28463099d17f727d7fe499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.590291 895 tablet_service.cc:1467] Processing CreateTablet for tablet 09d8d1e75d334dc5b1c8febd66a81d02 (DEFAULT_TABLE table=default.Table_2414450f78c54a5da31248d94e856cbb [id=11042364bcb9428996f8019c602a4b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.590320 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d029b5e24a91485a989ff462a678c4a2 (DEFAULT_TABLE table=default.Table_21a5abc833a749e487f214e77ffb3938 [id=eac56f9087164d8c81c85cf9996724b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.590649 885 tablet_service.cc:1467] Processing CreateTablet for tablet d0d53a3d762e49cdb8da84cfe68ad379 (DEFAULT_TABLE table=default.Table_40809af1edea441a8e0c9402b04b0c74 [id=2f5d6cc531744bbf8a470cdb04af6f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.590826 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3badd902aac749c49b80af176be90309 (DEFAULT_TABLE table=default.Table_07fd41bf19254a88bc4d0169c627b3c2 [id=335e03e466a94fdb8ecf32dd22ea3e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.591248 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2bd365ec9d6640318cfafe056f41dbf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:06.591310 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b924c0bdfda24e3ca0a78e60a925bcd1 (DEFAULT_TABLE table=default.Table_6da66d9b734d4dfc848a5ae9e5e184dd [id=e31b1a30b94f4f7da3ff4b6112445435]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:06.591681 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bd365ec9d6640318cfafe056f41dbf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.591718 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a2fb9168f0c74525807bc72b888cb35a (DEFAULT_TABLE table=default.Table_db4cd004494744f892fc049ed4e65a4e [id=18a9e0fa4a28463099d17f727d7fe499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.593151 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d0d53a3d762e49cdb8da84cfe68ad379 (DEFAULT_TABLE table=default.Table_40809af1edea441a8e0c9402b04b0c74 [id=2f5d6cc531744bbf8a470cdb04af6f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.593209 894 tablet_service.cc:1467] Processing CreateTablet for tablet bd8fdc90e32a47e19bdabd30fe2b3525 (DEFAULT_TABLE table=default.Table_190f7022156a45b0b2905f93a7f274df [id=b4d0791390ce4968ab49af9b242d57da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.593518 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 09d8d1e75d334dc5b1c8febd66a81d02 (DEFAULT_TABLE table=default.Table_2414450f78c54a5da31248d94e856cbb [id=11042364bcb9428996f8019c602a4b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.593571 892 tablet_service.cc:1467] Processing CreateTablet for tablet 14ce71f6c3c443a0862658ab3b7510e2 (DEFAULT_TABLE table=default.Table_b2efe8b52c934acaac976ac5af84531e [id=becef7430b45418aa8d60851bb2551a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.593953 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7a9f34b8d72242458260fc2cc645f85f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.594455 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a9f34b8d72242458260fc2cc645f85f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.595395 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 3badd902aac749c49b80af176be90309 (DEFAULT_TABLE table=default.Table_07fd41bf19254a88bc4d0169c627b3c2 [id=335e03e466a94fdb8ecf32dd22ea3e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.595656 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 14ce71f6c3c443a0862658ab3b7510e2 (DEFAULT_TABLE table=default.Table_b2efe8b52c934acaac976ac5af84531e [id=becef7430b45418aa8d60851bb2551a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.595834 1029 tablet_service.cc:1467] Processing CreateTablet for tablet bd8fdc90e32a47e19bdabd30fe2b3525 (DEFAULT_TABLE table=default.Table_190f7022156a45b0b2905f93a7f274df [id=b4d0791390ce4968ab49af9b242d57da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.598675 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_26f34d2971ef4d2291e2d02e53d39461" } modify_external_catalogs: true
W20250109 20:53:06.599680 854 leader_election.cc:343] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 7da1f3cafffe4f7ca39624b43831689e
I20250109 20:53:06.599794 854 leader_election.cc:304] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.600265 2552 raft_consensus.cc:2743] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.600858 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3b054fb299c84cf49c242e3ccb44daf8" } new_table_name: "default.Table_7a5615c2ebe34bd28be1ef292c7dd1c3" modify_external_catalogs: true
W20250109 20:53:06.605345 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 908781d94a054386b7d7c06c2c9fa3ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 908781d94a054386b7d7c06c2c9fa3ca already in progress: creating tablet
I20250109 20:53:06.559823 1158 tablet_bootstrap.cc:492] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:06.577957 746 tablet_service.cc:1467] Processing CreateTablet for tablet 4c3d64693f594ea7b367f5add0068aaf (DEFAULT_TABLE table=default.Table_3eb37633d0cd466a9e79e8fa50d54f03 [id=bbcf1ce2703d435e8bd783f22bf71683]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.580773 742 tablet_service.cc:1467] Processing CreateTablet for tablet 64427614c0ac40afb54343277cc578bb (DEFAULT_TABLE table=default.Table_a4bd298f3639422487e396ef57dd2756 [id=4d6c8c4ea93c4b7abd749a013462ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.582648 747 tablet_service.cc:1467] Processing CreateTablet for tablet 4a39059d6b274f35a4a076611c7fb828 (DEFAULT_TABLE table=default.Table_591e8768478a4bd28f197e2b94197c39 [id=7ea97514ee88484dbf97f4a0ec6e584d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.585251 757 tablet_service.cc:1467] Processing CreateTablet for tablet d029b5e24a91485a989ff462a678c4a2 (DEFAULT_TABLE table=default.Table_21a5abc833a749e487f214e77ffb3938 [id=eac56f9087164d8c81c85cf9996724b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.594237 755 tablet_service.cc:1467] Processing CreateTablet for tablet b924c0bdfda24e3ca0a78e60a925bcd1 (DEFAULT_TABLE table=default.Table_6da66d9b734d4dfc848a5ae9e5e184dd [id=e31b1a30b94f4f7da3ff4b6112445435]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.594482 740 tablet_service.cc:1467] Processing CreateTablet for tablet 09d8d1e75d334dc5b1c8febd66a81d02 (DEFAULT_TABLE table=default.Table_2414450f78c54a5da31248d94e856cbb [id=11042364bcb9428996f8019c602a4b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.594565 750 tablet_service.cc:1467] Processing CreateTablet for tablet a2fb9168f0c74525807bc72b888cb35a (DEFAULT_TABLE table=default.Table_db4cd004494744f892fc049ed4e65a4e [id=18a9e0fa4a28463099d17f727d7fe499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.594826 744 tablet_service.cc:1467] Processing CreateTablet for tablet d0d53a3d762e49cdb8da84cfe68ad379 (DEFAULT_TABLE table=default.Table_40809af1edea441a8e0c9402b04b0c74 [id=2f5d6cc531744bbf8a470cdb04af6f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.596268 741 tablet_service.cc:1467] Processing CreateTablet for tablet 3badd902aac749c49b80af176be90309 (DEFAULT_TABLE table=default.Table_07fd41bf19254a88bc4d0169c627b3c2 [id=335e03e466a94fdb8ecf32dd22ea3e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.596563 749 tablet_service.cc:1467] Processing CreateTablet for tablet 14ce71f6c3c443a0862658ab3b7510e2 (DEFAULT_TABLE table=default.Table_b2efe8b52c934acaac976ac5af84531e [id=becef7430b45418aa8d60851bb2551a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.596612 739 tablet_service.cc:1467] Processing CreateTablet for tablet bd8fdc90e32a47e19bdabd30fe2b3525 (DEFAULT_TABLE table=default.Table_190f7022156a45b0b2905f93a7f274df [id=b4d0791390ce4968ab49af9b242d57da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:06.606365 1158 ts_tablet_manager.cc:1397] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.114s user 0.002s sys 0.002s
I20250109 20:53:06.607301 1158 raft_consensus.cc:357] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.607615 1158 raft_consensus.cc:383] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.607707 1158 raft_consensus.cc:738] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.607964 1158 consensus_queue.cc:260] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.609867 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c01a7fe9ae3947a59c88c37c98a59cab" } modify_external_catalogs: true
I20250109 20:53:06.610013 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c01a7fe9ae3947a59c88c37c98a59cab" } modify_external_catalogs: true
I20250109 20:53:06.610600 1158 ts_tablet_manager.cc:1428] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:06.610953 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_26f34d2971ef4d2291e2d02e53d39461" } modify_external_catalogs: true
I20250109 20:53:06.611227 778 raft_consensus.cc:1270] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Refusing update from remote peer 5fcc647ed479491b88e5ffacf2927a1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250109 20:53:06.611727 2475 consensus_queue.cc:1035] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:06.618753 2413 raft_consensus.cc:491] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.618948 2413 raft_consensus.cc:513] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.619483 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3d64693f594ea7b367f5add0068aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.619532 2413 leader_election.cc:290] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.623821 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64427614c0ac40afb54343277cc578bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.624068 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ce71f6c3c443a0862658ab3b7510e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.624413 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d8d1e75d334dc5b1c8febd66a81d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.624554 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d53a3d762e49cdb8da84cfe68ad379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.624732 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b924c0bdfda24e3ca0a78e60a925bcd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.624903 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3badd902aac749c49b80af176be90309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.625049 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d029b5e24a91485a989ff462a678c4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.625203 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a39059d6b274f35a4a076611c7fb828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.626127 743 tablet_service.cc:1514] Processing DeleteTablet for tablet bd62a3ae64a143998bb1172b3a97c69f with delete_type TABLET_DATA_DELETED (replaced by tablet 575f3cb9f61143ab949fa85f47edcafb) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.629140 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fb9168f0c74525807bc72b888cb35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.629216 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4abc19297e4029b1b7c5396b49b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.636927 2546 raft_consensus.cc:491] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.637058 2546 raft_consensus.cc:513] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.637603 2546 leader_election.cc:290] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:06.637703 2546 raft_consensus.cc:491] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.637781 2546 raft_consensus.cc:513] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.638192 2546 leader_election.cc:290] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:06.639022 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45cd5ffe7040e385d8cdfb97ed011a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:06.639220 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f4bdda16554d9d81de5ca5c5205f4c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:06.639487 990 leader_election.cc:343] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.639554 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8fdc90e32a47e19bdabd30fe2b3525. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:06.639765 990 leader_election.cc:343] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.645999 1166 ts_tablet_manager.cc:1918] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:06.646104 1166 log.cc:1198] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6c4abc19297e4029b1b7c5396b49b967
I20250109 20:53:06.646378 1166 ts_tablet_manager.cc:1939] T 6c4abc19297e4029b1b7c5396b49b967 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:06.651120 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3badd902aac749c49b80af176be90309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.651165 1160 ts_tablet_manager.cc:1918] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:06.651268 1160 log.cc:1198] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/16c9f7274a7542e89a19bd4afcf89205
I20250109 20:53:06.651297 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d53a3d762e49cdb8da84cfe68ad379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.651579 1160 ts_tablet_manager.cc:1939] T 16c9f7274a7542e89a19bd4afcf89205 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:06.652110 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3d64693f594ea7b367f5add0068aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.652467 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a39059d6b274f35a4a076611c7fb828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.652786 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64427614c0ac40afb54343277cc578bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.653101 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d029b5e24a91485a989ff462a678c4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.653455 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b924c0bdfda24e3ca0a78e60a925bcd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.653770 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fb9168f0c74525807bc72b888cb35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.654085 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d8d1e75d334dc5b1c8febd66a81d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.654390 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8fdc90e32a47e19bdabd30fe2b3525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.654682 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ce71f6c3c443a0862658ab3b7510e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.659039 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd62a3ae64a143998bb1172b3a97c69f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:06.659559 712 leader_election.cc:343] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.659933 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3d64693f594ea7b367f5add0068aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.660100 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8fdc90e32a47e19bdabd30fe2b3525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.660378 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3badd902aac749c49b80af176be90309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.660741 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fb9168f0c74525807bc72b888cb35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.660604 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7da583c325ba439aa802225a7342b40b" } modify_external_catalogs: true
I20250109 20:53:06.660840 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7da583c325ba439aa802225a7342b40b" } modify_external_catalogs: true
I20250109 20:53:06.661083 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d53a3d762e49cdb8da84cfe68ad379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.661465 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14ce71f6c3c443a0862658ab3b7510e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.661823 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b924c0bdfda24e3ca0a78e60a925bcd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.662153 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a39059d6b274f35a4a076611c7fb828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.662484 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d029b5e24a91485a989ff462a678c4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.662860 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d8d1e75d334dc5b1c8febd66a81d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.663784 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1b0e82dbd939465ca7044f0aa1f46f06
I20250109 20:53:06.664502 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_02829bc5be564d358bcca8febc5c3bd5" } new_table_name: "default.Table_e94f4f4106534d17a1a10004d35f8814" modify_external_catalogs: true
I20250109 20:53:06.665933 1158 tablet_bootstrap.cc:492] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:06.666441 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64427614c0ac40afb54343277cc578bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:06.666458 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0e82dbd939465ca7044f0aa1f46f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.668426 1158 tablet_bootstrap.cc:654] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.669339 1157 tablet_replica.cc:331] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:06.669617 1157 raft_consensus.cc:2238] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:06.672001 1157 raft_consensus.cc:2267] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:06.681412 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45cd5ffe7040e385d8cdfb97ed011a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:06.684892 1157 ts_tablet_manager.cc:1905] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:06.685402 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f4bdda16554d9d81de5ca5c5205f4c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:06.702621 1165 tablet_bootstrap.cc:492] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.702759 1165 ts_tablet_manager.cc:1397] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.152s user 0.016s sys 0.015s
I20250109 20:53:06.703359 1165 raft_consensus.cc:357] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.703527 1165 raft_consensus.cc:383] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.703583 1165 raft_consensus.cc:738] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.703796 1165 consensus_queue.cc:260] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.704305 1165 ts_tablet_manager.cc:1428] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:06.705469 1165 tablet_bootstrap.cc:492] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:06.707839 1165 tablet_bootstrap.cc:654] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.709499 1176 tablet_bootstrap.cc:492] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.709640 1176 ts_tablet_manager.cc:1397] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.133s user 0.019s sys 0.010s
I20250109 20:53:06.721691 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0e82dbd939465ca7044f0aa1f46f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.722595 1165 tablet_bootstrap.cc:492] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.722707 1165 ts_tablet_manager.cc:1397] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250109 20:53:06.723403 1165 raft_consensus.cc:357] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.723583 1165 raft_consensus.cc:383] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.723623 1165 raft_consensus.cc:738] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.723783 1165 consensus_queue.cc:260] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.724236 1165 ts_tablet_manager.cc:1428] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:06.738337 881 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0d889cb1114edc8b799b41b23a54a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:46 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.741873 2475 raft_consensus.cc:491] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.742038 2475 raft_consensus.cc:513] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.742671 2475 leader_election.cc:290] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:53:06.743319 989 consensus_peers.cc:487] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:06.743945 2552 raft_consensus.cc:491] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.744117 2552 raft_consensus.cc:513] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.744894 2552 leader_election.cc:290] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.747144 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1186) took 2880 ms. Trace:
I20250109 20:53:06.747246 1160 rpcz_store.cc:276] 0109 20:53:03.866222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.866390 (+ 168us) service_pool.cc:225] Handling call
0109 20:53:06.277453 (+2411063us) ts_tablet_manager.cc:1145] Deleting tablet 16c9f7274a7542e89a19bd4afcf89205
0109 20:53:06.277462 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.468878 (+191416us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.651048 (+182170us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.747114 (+ 96066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2410882}
I20250109 20:53:06.747684 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1183) took 2970 ms. Trace:
I20250109 20:53:06.747627 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3911b54a040144baadfc4e0088d58e64" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:06.747740 1160 rpcz_store.cc:276] 0109 20:53:03.777089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.870678 (+ 93589us) service_pool.cc:225] Handling call
0109 20:53:06.747580 (+2876902us) ts_tablet_manager.cc:1145] Deleting tablet c6e4cfb5a07841828e5206d100b4f835
0109 20:53:06.747588 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.747671 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2876630}
I20250109 20:53:06.748020 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1188) took 2839 ms. Trace:
I20250109 20:53:06.748096 1160 rpcz_store.cc:276] 0109 20:53:03.908506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.908548 (+ 42us) service_pool.cc:225] Handling call
0109 20:53:06.747911 (+2839363us) ts_tablet_manager.cc:1145] Deleting tablet 21dbbfa0af5b4f0487dd5d9db8efaeda
0109 20:53:06.747922 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.748004 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2839133}
I20250109 20:53:06.748406 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1189) took 2836 ms. Trace:
I20250109 20:53:06.748471 1160 rpcz_store.cc:276] 0109 20:53:03.911920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.911963 (+ 43us) service_pool.cc:225] Handling call
0109 20:53:06.748314 (+2836351us) ts_tablet_manager.cc:1145] Deleting tablet 7b8344dbb1c540abadd22531e19a52d1
0109 20:53:06.748324 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.748392 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2836208}
I20250109 20:53:06.748868 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1190) took 2834 ms. Trace:
I20250109 20:53:06.748947 1160 rpcz_store.cc:276] 0109 20:53:03.914802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.914846 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:06.748656 (+2833810us) ts_tablet_manager.cc:1145] Deleting tablet 6c4abc19297e4029b1b7c5396b49b967
0109 20:53:06.748664 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.748850 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2833684}
I20250109 20:53:06.749271 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1187) took 2840 ms. Trace:
I20250109 20:53:06.749341 1160 rpcz_store.cc:276] 0109 20:53:03.908405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.926667 (+ 18262us) service_pool.cc:225] Handling call
0109 20:53:06.749159 (+2822492us) ts_tablet_manager.cc:1145] Deleting tablet f0d09d4894634825ad3aa7933bfb6010
0109 20:53:06.749168 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.749257 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2822348}
I20250109 20:53:06.749646 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1191) took 2774 ms. Trace:
I20250109 20:53:06.749711 1160 rpcz_store.cc:276] 0109 20:53:03.974949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.975001 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:06.749542 (+2774541us) ts_tablet_manager.cc:1145] Deleting tablet e1faf5d79a9b43ffafc9c34a5937fcc0
0109 20:53:06.749551 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.749634 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774363}
I20250109 20:53:06.750029 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1192) took 2769 ms. Trace:
I20250109 20:53:06.750097 1160 rpcz_store.cc:276] 0109 20:53:03.980857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.980905 (+ 48us) service_pool.cc:225] Handling call
0109 20:53:06.749927 (+2769022us) ts_tablet_manager.cc:1145] Deleting tablet 8f2ba81d736745fb9d56fd9a7c229153
0109 20:53:06.749936 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.750017 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2768859}
I20250109 20:53:06.750321 1160 tablet_replica.cc:331] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:06.750515 1160 raft_consensus.cc:2238] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:06.750648 1160 raft_consensus.cc:2267] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:06.751668 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 16c9f7274a7542e89a19bd4afcf89205 (table default.Table_455cff8c7fb2449f908ca21e7da023af [id=32b7c370648b4318be6d2e2c0284e9d7]) successfully deleted
W20250109 20:53:06.751928 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c6e4cfb5a07841828e5206d100b4f835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6e4cfb5a07841828e5206d100b4f835 already in progress: creating tablet
W20250109 20:53:06.752135 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21dbbfa0af5b4f0487dd5d9db8efaeda already in progress: creating tablet
W20250109 20:53:06.752287 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7b8344dbb1c540abadd22531e19a52d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b8344dbb1c540abadd22531e19a52d1 already in progress: creating tablet
W20250109 20:53:06.752432 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6c4abc19297e4029b1b7c5396b49b967 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c4abc19297e4029b1b7c5396b49b967 already in progress: creating tablet
W20250109 20:53:06.752594 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f0d09d4894634825ad3aa7933bfb6010 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0d09d4894634825ad3aa7933bfb6010 already in progress: creating tablet
W20250109 20:53:06.752818 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1faf5d79a9b43ffafc9c34a5937fcc0 already in progress: creating tablet
I20250109 20:53:06.753336 1160 ts_tablet_manager.cc:1905] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:06.754717 1165 tablet_bootstrap.cc:492] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:06.757496 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0e82dbd939465ca7044f0aa1f46f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.772038 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1a734623b43b4fab8389a393281e3a3f" } modify_external_catalogs: true
I20250109 20:53:06.772297 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1a734623b43b4fab8389a393281e3a3f" } modify_external_catalogs: true
I20250109 20:53:06.774266 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:06.774471 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 74f4bdda16554d9d81de5ca5c5205f4c (table default.Table_d2d8f346b1d2494cb588d606237080a0 [id=dfdbf927d4ff4f329975148ef22d8940]). Expected version 1 got 0
I20250109 20:53:06.774600 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 8a45cd5ffe7040e385d8cdfb97ed011a (table default.Table_b66f03a4dc5241978f3ef4e00ebcd0a0 [id=4d3659cafa3246f1a84d6a4ddd44f82f]). Expected version 3 got 0
I20250109 20:53:06.776188 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_31d3d34c2b5347fbb18b9b7d7e034d5b" } new_table_name: "default.Table_26d7b2a208ea4ffabd23868fb172ee35" modify_external_catalogs: true
W20250109 20:53:06.776953 854 leader_election.cc:343] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: a0d09daa2eed41c3952fa21aa2cdd09e
W20250109 20:53:06.777585 854 leader_election.cc:343] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: a0d09daa2eed41c3952fa21aa2cdd09e
I20250109 20:53:06.777669 854 leader_election.cc:304] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.778070 2552 raft_consensus.cc:2743] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.779225 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_be95c280cf364cbeab60808f1d88975a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.779718 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be95c280cf364cbeab60808f1d88975a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.781006 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f114cef231344fd4a0a4c9e46ae277e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.781448 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f114cef231344fd4a0a4c9e46ae277e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.782276 2413 raft_consensus.cc:491] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.782413 2413 raft_consensus.cc:513] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.782636 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 048be84b36a1450a818f05fb36cbf3b9
I20250109 20:53:06.783008 2413 leader_election.cc:290] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.783210 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f5ff906ba8ba4efd971aa6bc36593cf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:06.783550 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048be84b36a1450a818f05fb36cbf3b9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:06.783674 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5ff906ba8ba4efd971aa6bc36593cf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:06.783998 712 leader_election.cc:343] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.747812 1176 raft_consensus.cc:357] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.784482 1176 raft_consensus.cc:383] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250109 20:53:06.784547 989 leader_election.cc:343] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.784636 989 leader_election.cc:304] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:06.750222 990 leader_election.cc:343] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.784935 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bc32ce3dff0f4d18bb17c6784b0bab41
I20250109 20:53:06.785441 2475 raft_consensus.cc:2743] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.784551 1176 raft_consensus.cc:738] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.786132 1176 consensus_queue.cc:260] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.786655 1176 ts_tablet_manager.cc:1428] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.077s user 0.001s sys 0.001s
W20250109 20:53:06.788321 989 leader_election.cc:343] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.788668 989 leader_election.cc:304] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.790038 2546 raft_consensus.cc:2743] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.790278 749 tablet_service.cc:1514] Processing DeleteTablet for tablet bc32ce3dff0f4d18bb17c6784b0bab41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.792125 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 048be84b36a1450a818f05fb36cbf3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.793046 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet bc32ce3dff0f4d18bb17c6784b0bab41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.796849 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e with delete_type TABLET_DATA_DELETED (replaced by tablet 37ee437300414eccba9926e824f72f27) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.797075 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c83e709a8d4e078e67af58ec51beac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.797652 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1226) took 2883 ms. Trace:
I20250109 20:53:06.797753 1166 rpcz_store.cc:276] 0109 20:53:03.913791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.913843 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:06.453961 (+2540118us) ts_tablet_manager.cc:1145] Deleting tablet 6c4abc19297e4029b1b7c5396b49b967
0109 20:53:06.453971 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.551858 (+ 97887us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.645906 (+ 94048us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.797632 (+151726us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2539993}
I20250109 20:53:06.798143 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1223) took 2889 ms. Trace:
I20250109 20:53:06.798223 1166 rpcz_store.cc:276] 0109 20:53:03.908975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.926855 (+ 17880us) service_pool.cc:225] Handling call
0109 20:53:06.798019 (+2871164us) ts_tablet_manager.cc:1145] Deleting tablet f0d09d4894634825ad3aa7933bfb6010
0109 20:53:06.798027 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.798129 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2870972}
I20250109 20:53:06.798393 1176 tablet_bootstrap.cc:492] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.798522 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1225) took 2886 ms. Trace:
I20250109 20:53:06.798593 1166 rpcz_store.cc:276] 0109 20:53:03.912448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.913787 (+ 1339us) service_pool.cc:225] Handling call
0109 20:53:06.798424 (+2884637us) ts_tablet_manager.cc:1145] Deleting tablet 7b8344dbb1c540abadd22531e19a52d1
0109 20:53:06.798434 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.798508 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871074}
I20250109 20:53:06.798884 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1224) took 2889 ms. Trace:
I20250109 20:53:06.798954 1166 rpcz_store.cc:276] 0109 20:53:03.909089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.927386 (+ 18297us) service_pool.cc:225] Handling call
0109 20:53:06.798800 (+2871414us) ts_tablet_manager.cc:1145] Deleting tablet 21dbbfa0af5b4f0487dd5d9db8efaeda
0109 20:53:06.798807 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.798870 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871244}
I20250109 20:53:06.799446 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1227) took 2821 ms. Trace:
I20250109 20:53:06.799528 1166 rpcz_store.cc:276] 0109 20:53:03.977671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.977718 (+ 47us) service_pool.cc:225] Handling call
0109 20:53:06.799351 (+2821633us) ts_tablet_manager.cc:1145] Deleting tablet e1faf5d79a9b43ffafc9c34a5937fcc0
0109 20:53:06.799359 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.799430 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2821474}
I20250109 20:53:06.799811 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1228) took 2818 ms. Trace:
I20250109 20:53:06.799880 1166 rpcz_store.cc:276] 0109 20:53:03.981249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:03.981305 (+ 56us) service_pool.cc:225] Handling call
0109 20:53:06.799727 (+2818422us) ts_tablet_manager.cc:1145] Deleting tablet 8f2ba81d736745fb9d56fd9a7c229153
0109 20:53:06.799735 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.799797 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2818281}
I20250109 20:53:06.800048 1176 tablet_bootstrap.cc:654] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.800297 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6c4abc19297e4029b1b7c5396b49b967 (table default.Table_eff4d27a17b3425c92c3d388ee1cc73e [id=6f77cddda4de4f6083cedc70e3026b6d]) successfully deleted
I20250109 20:53:06.800856 883 tablet_service.cc:1514] Processing DeleteTablet for tablet bc32ce3dff0f4d18bb17c6784b0bab41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:06.800825 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f0d09d4894634825ad3aa7933bfb6010 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0d09d4894634825ad3aa7933bfb6010 already in progress: creating tablet
I20250109 20:53:06.801123 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 048be84b36a1450a818f05fb36cbf3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:06.801290 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7b8344dbb1c540abadd22531e19a52d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b8344dbb1c540abadd22531e19a52d1 already in progress: creating tablet
W20250109 20:53:06.801569 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 21dbbfa0af5b4f0487dd5d9db8efaeda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21dbbfa0af5b4f0487dd5d9db8efaeda already in progress: creating tablet
I20250109 20:53:06.801687 890 tablet_service.cc:1514] Processing DeleteTablet for tablet e16a1d9a5b374b8dbf725eb32ffadb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:06.801982 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e1faf5d79a9b43ffafc9c34a5937fcc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1faf5d79a9b43ffafc9c34a5937fcc0 already in progress: creating tablet
W20250109 20:53:06.802304 713 leader_election.cc:343] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: bd62a3ae64a143998bb1172b3a97c69f
I20250109 20:53:06.802417 713 leader_election.cc:304] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:06.801574 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a5f945b345d4c029798756e63867156 consensus state has no leader
I20250109 20:53:06.802661 2413 raft_consensus.cc:2743] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.802714 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 048be84b36a1450a818f05fb36cbf3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.803435 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3911b54a040144baadfc4e0088d58e64" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:06.804396 2556 tablet.cc:1721] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:06.814018 1158 tablet_bootstrap.cc:492] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:06.814354 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7771b3421a9449929095a6e25dcbf470" } new_table_name: "default.Table_333a08dce1d74059a4a06ec1e7eb7513" modify_external_catalogs: true
I20250109 20:53:06.817003 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f4204ccf6eb1426192463345ae7109a4" } new_table_name: "default.Table_8bc289937bd247a1a05ac1f621324338" modify_external_catalogs: true
I20250109 20:53:06.819685 1158 ts_tablet_manager.cc:1397] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.154s user 0.004s sys 0.000s
I20250109 20:53:06.821261 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_db5cb09bc1544977b3d40680b347ac2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.821720 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db5cb09bc1544977b3d40680b347ac2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.821923 1158 raft_consensus.cc:357] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.822093 1158 raft_consensus.cc:383] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.822175 1158 raft_consensus.cc:738] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.822732 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048be84b36a1450a818f05fb36cbf3b9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:06.823284 1053 raft_consensus.cc:2463] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:06.826362 713 leader_election.cc:304] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.826625 2534 raft_consensus.cc:2798] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:06.826715 2534 raft_consensus.cc:491] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:06.826792 2534 raft_consensus.cc:3054] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:06.827237 1176 tablet_bootstrap.cc:492] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.827508 1176 ts_tablet_manager.cc:1397] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.010s
I20250109 20:53:06.829668 1176 raft_consensus.cc:357] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.830211 1176 raft_consensus.cc:383] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.830924 1176 raft_consensus.cc:738] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.832700 1176 consensus_queue.cc:260] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.833222 1176 ts_tablet_manager.cc:1428] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:53:06.837340 1165 tablet_bootstrap.cc:654] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.840328 2552 raft_consensus.cc:491] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.840446 2552 raft_consensus.cc:513] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.841054 2552 leader_election.cc:290] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:06.842198 854 leader_election.cc:343] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 2a4f047e8fa34ca79e78915c61a8ba12
W20250109 20:53:06.843149 989 leader_election.cc:343] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.843354 989 leader_election.cc:304] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.843827 2368 raft_consensus.cc:2743] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:06.848079 854 leader_election.cc:343] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2a4f047e8fa34ca79e78915c61a8ba12
I20250109 20:53:06.848176 854 leader_election.cc:304] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.848783 2552 raft_consensus.cc:2743] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.851011 2534 raft_consensus.cc:513] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.851627 2534 leader_election.cc:290] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.856074 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048be84b36a1450a818f05fb36cbf3b9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:06.856664 712 leader_election.cc:343] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:06.858559 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 622cc5f673694546a594f968621f1011 consensus state has no leader
I20250109 20:53:06.858162 2558 tablet.cc:1721] T 4f3c276fab6a4d068d64ea5d23c9a595 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:06.860472 1158 consensus_queue.cc:260] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.861164 1158 ts_tablet_manager.cc:1428] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.040s user 0.001s sys 0.000s
I20250109 20:53:06.861575 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4597aa225a374487b13647961b1843d8
I20250109 20:53:06.862372 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_350b1b2845b4456994e4e3e61ea7fd0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:06.862869 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_350b1b2845b4456994e4e3e61ea7fd0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.863750 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 4597aa225a374487b13647961b1843d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.864730 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4597aa225a374487b13647961b1843d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:06.865803 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d0e5377172f478cbefcc978ed2a9e0d consensus state has no leader
I20250109 20:53:06.871582 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6cb88771d2b148bdae6e427c1b25b9a9
I20250109 20:53:06.872532 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb88771d2b148bdae6e427c1b25b9a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:06.873162 1166 tablet_replica.cc:331] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:06.873399 1166 raft_consensus.cc:2238] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:06.865896 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048be84b36a1450a818f05fb36cbf3b9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:06.873565 1166 raft_consensus.cc:2267] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:06.873631 1051 raft_consensus.cc:3054] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:06.874122 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4597aa225a374487b13647961b1843d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.874123 1166 ts_tablet_manager.cc:1905] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:06.875052 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb88771d2b148bdae6e427c1b25b9a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:06.875388 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb88771d2b148bdae6e427c1b25b9a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:06.876579 453 master_service.cc:946] ReplaceTablet: received request to replace tablet beca5f1ed9d3434fb7de90cd74a6f09b from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:06.876964 1176 tablet_bootstrap.cc:492] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.879079 2600 tablet.cc:1721] T 4f3c276fab6a4d068d64ea5d23c9a595 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:06.881688 1165 tablet_bootstrap.cc:492] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.881815 1165 ts_tablet_manager.cc:1397] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.010s
I20250109 20:53:06.882509 1165 raft_consensus.cc:357] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.882687 1165 raft_consensus.cc:383] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.882743 1165 raft_consensus.cc:738] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.882970 1165 consensus_queue.cc:260] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.884099 964 heartbeater.cc:346] Connected to a master server at 127.0.104.60:41583
I20250109 20:53:06.885635 1165 ts_tablet_manager.cc:1428] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250109 20:53:06.889932 2599 tablet.cc:1721] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:06.894527 1176 tablet_bootstrap.cc:654] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.899523 2368 raft_consensus.cc:491] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.899629 2368 raft_consensus.cc:513] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.900288 2368 leader_election.cc:290] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:06.901276 1165 tablet_bootstrap.cc:492] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:06.901875 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2fcddba2ac4d82b2d25bfc64a5967e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:06.902407 990 leader_election.cc:343] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.903460 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2fcddba2ac4d82b2d25bfc64a5967e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:06.905054 989 leader_election.cc:343] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.907248 989 leader_election.cc:304] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.907718 2546 raft_consensus.cc:2743] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.912634 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f7e3652017946d481090b9940d32e97 from {username='slave'} at 127.0.0.1:33768
W20250109 20:53:06.916188 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a13b6eb4563b46aeb6a89faae823f62a consensus state has no leader
I20250109 20:53:06.932483 876 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57646 (request call id 188) took 1150 ms. Trace:
I20250109 20:53:06.932686 876 rpcz_store.cc:276] 0109 20:53:05.781671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.785342 (+ 3671us) service_pool.cc:225] Handling call
0109 20:53:06.932455 (+1147113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250109 20:53:06.942704 1165 tablet_bootstrap.cc:654] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.948490 1166 ts_tablet_manager.cc:1918] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:06.948606 1166 log.cc:1198] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c865faed94c3493ea638826bdcd4253d
I20250109 20:53:06.948968 1166 ts_tablet_manager.cc:1939] T c865faed94c3493ea638826bdcd4253d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:06.951936 1051 raft_consensus.cc:2463] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:06.959530 2413 raft_consensus.cc:491] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:06.959692 2413 raft_consensus.cc:513] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.960144 1176 tablet_bootstrap.cc:492] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:06.960294 1176 ts_tablet_manager.cc:1397] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.006s
I20250109 20:53:06.960501 2413 leader_election.cc:290] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:06.961024 1176 raft_consensus.cc:357] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.961192 1176 raft_consensus.cc:383] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.961241 1176 raft_consensus.cc:738] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.961423 1176 consensus_queue.cc:260] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.961905 1176 ts_tablet_manager.cc:1428] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:06.962038 1160 ts_tablet_manager.cc:1918] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:06.962141 1160 log.cc:1198] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/73e82535780c47ca8b9cd94b013b4b42
I20250109 20:53:06.962164 1176 tablet_bootstrap.cc:492] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:06.962436 1160 ts_tablet_manager.cc:1939] T 73e82535780c47ca8b9cd94b013b4b42 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:06.964166 1176 tablet_bootstrap.cc:654] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:06.966964 713 leader_election.cc:304] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:06.967779 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d80f0ae2bf3e419faf22aa448e220dfd" } modify_external_catalogs: true
I20250109 20:53:06.967944 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d80f0ae2bf3e419faf22aa448e220dfd" } modify_external_catalogs: true
I20250109 20:53:06.968859 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6ecacbb0b64b49938e20e76653b50f4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:06.969138 2413 raft_consensus.cc:2798] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:06.969264 2413 raft_consensus.cc:695] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
W20250109 20:53:06.969480 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ecacbb0b64b49938e20e76653b50f4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:06.969483 2413 consensus_queue.cc:237] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:06.971475 713 leader_election.cc:343] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 6c4abc19297e4029b1b7c5396b49b967
I20250109 20:53:06.974251 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4abc19297e4029b1b7c5396b49b967" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:06.974856 712 leader_election.cc:343] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:06.974953 712 leader_election.cc:304] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:06.975376 2534 raft_consensus.cc:2743] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:06.994432 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1231) took 2848 ms. Trace:
I20250109 20:53:06.994547 1166 rpcz_store.cc:276] 0109 20:53:04.145657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.145709 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:06.800086 (+2654377us) ts_tablet_manager.cc:1145] Deleting tablet c865faed94c3493ea638826bdcd4253d
0109 20:53:06.873074 (+ 72988us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.945245 (+ 72171us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.948390 (+ 3145us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.994412 (+ 46022us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654152}
I20250109 20:53:06.994875 1166 tablet_replica.cc:331] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:06.995054 1166 raft_consensus.cc:2238] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:06.995041 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet c865faed94c3493ea638826bdcd4253d (table default.Table_c07faa9c0a64412588c149655620702d [id=be7028b0c99c4883aded712bdd660cbb]) successfully deleted
I20250109 20:53:06.995201 1166 raft_consensus.cc:2267] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:06.995776 1166 ts_tablet_manager.cc:1905] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:06.997368 1165 tablet_bootstrap.cc:492] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:06.997501 1165 ts_tablet_manager.cc:1397] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.003s
I20250109 20:53:06.998148 1165 raft_consensus.cc:357] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:06.998383 1165 raft_consensus.cc:383] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:06.998443 1165 raft_consensus.cc:738] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:06.998636 1165 consensus_queue.cc:260] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.002593 2556 tablet.cc:1721] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:07.003657 2597 tablet.cc:1721] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:07.006799 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9234d242e6e44e3494df97820a63cc88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.007301 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9234d242e6e44e3494df97820a63cc88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.007465 1165 ts_tablet_manager.cc:1428] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250109 20:53:07.010769 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 825220629faa4d2481e0db9fa226c306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:58 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.011992 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e2a32f40c0aa43d8bfb4cc97d297fc76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.012133 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f3b6f90f78b545b7973d76289c9a0413"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.012477 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2a32f40c0aa43d8bfb4cc97d297fc76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:07.012549 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3b6f90f78b545b7973d76289c9a0413 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.013594 1165 tablet_bootstrap.cc:492] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.013554 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_38713c5cbbbf44ba95a5b4710cdfcd44" } modify_external_catalogs: true
I20250109 20:53:07.013672 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_38713c5cbbbf44ba95a5b4710cdfcd44" } modify_external_catalogs: true
I20250109 20:53:07.014025 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b5e396ef02d74355a6735e5095163c92" } modify_external_catalogs: true
I20250109 20:53:07.014150 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b5e396ef02d74355a6735e5095163c92" } modify_external_catalogs: true
I20250109 20:53:07.014564 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0a8b24b301ef44c686368ec4f0ea1714"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.014951 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a8b24b301ef44c686368ec4f0ea1714 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.015205 1165 tablet_bootstrap.cc:654] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.016045 2368 raft_consensus.cc:491] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.016165 2368 raft_consensus.cc:513] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.016767 2368 leader_election.cc:290] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:07.017269 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2797e1df3d12476f89fb5155523c3935" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:07.017729 990 leader_election.cc:343] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.019590 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2797e1df3d12476f89fb5155523c3935" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.022318 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 50517.4us
I20250109 20:53:07.024039 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_21a5abc833a749e487f214e77ffb3938" } new_table_name: "default.Table_f076f56bae644c379334281db5a1e363" modify_external_catalogs: true
I20250109 20:53:07.029541 1165 tablet_bootstrap.cc:492] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.029688 1165 ts_tablet_manager.cc:1397] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20250109 20:53:07.029968 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 9d1c9f392439440bba88a86dbd6d853e from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:07.030403 1165 raft_consensus.cc:357] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.030606 1165 raft_consensus.cc:383] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.030668 1165 raft_consensus.cc:738] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.030848 1165 consensus_queue.cc:260] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.030927 2413 raft_consensus.cc:491] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.031029 2413 raft_consensus.cc:513] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.031841 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "712acab92e0f44a3a585d7d871498aa3" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.032299 712 leader_election.cc:343] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.032649 2413 leader_election.cc:290] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.036741 1157 ts_tablet_manager.cc:1918] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.036854 1157 log.cc:1198] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/85a95a31468b431fb4304cf6680016da
I20250109 20:53:07.036890 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 14fb1017fe6c4f5abd640ad1f6214232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:02 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.037119 1157 ts_tablet_manager.cc:1939] T 85a95a31468b431fb4304cf6680016da P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.037804 1176 tablet_bootstrap.cc:492] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:07.037940 1176 ts_tablet_manager.cc:1397] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250109 20:53:07.038668 1176 raft_consensus.cc:357] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.038859 1176 raft_consensus.cc:383] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.038928 1176 raft_consensus.cc:738] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.039134 1176 consensus_queue.cc:260] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.039628 1176 ts_tablet_manager.cc:1428] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:07.039881 1176 tablet_bootstrap.cc:492] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:07.039995 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1193) took 2924 ms. Trace:
I20250109 20:53:07.040089 1160 rpcz_store.cc:276] 0109 20:53:04.115045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.115096 (+ 51us) service_pool.cc:225] Handling call
0109 20:53:06.750282 (+2635186us) ts_tablet_manager.cc:1145] Deleting tablet 73e82535780c47ca8b9cd94b013b4b42
0109 20:53:06.750289 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.850488 (+100199us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:06.961946 (+111458us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.039980 (+ 78034us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2635003}
I20250109 20:53:07.040513 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1196) took 2889 ms. Trace:
I20250109 20:53:07.040601 1160 rpcz_store.cc:276] 0109 20:53:04.150780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.150826 (+ 46us) service_pool.cc:225] Handling call
0109 20:53:07.040365 (+2889539us) ts_tablet_manager.cc:1145] Deleting tablet 7357b4f67dc6448dbb8f77dce81731f3
0109 20:53:07.040373 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.040485 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2889406}
I20250109 20:53:07.040925 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1198) took 2888 ms. Trace:
I20250109 20:53:07.040999 1160 rpcz_store.cc:276] 0109 20:53:04.152088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.152235 (+ 147us) service_pool.cc:225] Handling call
0109 20:53:07.040800 (+2888565us) ts_tablet_manager.cc:1145] Deleting tablet cb880d4da1ca4c2c851fc8c1e0846ce4
0109 20:53:07.040808 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.040897 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2888402}
I20250109 20:53:07.041267 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1200) took 2876 ms. Trace:
W20250109 20:53:07.041422 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d029b5e24a91485a989ff462a678c4a2 consensus state has no leader
I20250109 20:53:07.041376 1160 rpcz_store.cc:276] 0109 20:53:04.164869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.164913 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:07.041165 (+2876252us) ts_tablet_manager.cc:1145] Deleting tablet d79437d800f44c5a8222eea7dc2cd4e3
0109 20:53:07.041172 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.041238 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2876101}
I20250109 20:53:07.041664 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1202) took 2866 ms. Trace:
I20250109 20:53:07.041729 1165 ts_tablet_manager.cc:1428] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:53:07.041766 1160 rpcz_store.cc:276] 0109 20:53:04.174697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.174743 (+ 46us) service_pool.cc:225] Handling call
0109 20:53:07.041549 (+2866806us) ts_tablet_manager.cc:1145] Deleting tablet ac867ce748f749d88885ef13654983ec
0109 20:53:07.041556 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.041634 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2866595}
I20250109 20:53:07.042757 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 73e82535780c47ca8b9cd94b013b4b42 (table default.Table_f7702e93eb894ea5bbf0af322f94c29c [id=0193ea72702947b590c27b63fd70f93f]) successfully deleted
W20250109 20:53:07.042984 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7357b4f67dc6448dbb8f77dce81731f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7357b4f67dc6448dbb8f77dce81731f3 already in progress: creating tablet
I20250109 20:53:07.043004 1165 tablet_bootstrap.cc:492] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:53:07.043161 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb880d4da1ca4c2c851fc8c1e0846ce4 already in progress: creating tablet
W20250109 20:53:07.043327 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d79437d800f44c5a8222eea7dc2cd4e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d79437d800f44c5a8222eea7dc2cd4e3 already in progress: creating tablet
W20250109 20:53:07.043484 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ac867ce748f749d88885ef13654983ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac867ce748f749d88885ef13654983ec already in progress: creating tablet
I20250109 20:53:07.044456 1165 tablet_bootstrap.cc:654] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.045017 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet f407b76780974cf7b3635fa1bf1585ec (table default.Table_6859465fc7f4440eaf4c387bc97b02ad [id=32d713e3cdee403cb9193769815a07fa]). Expected version 2 got 0
I20250109 20:53:07.045743 746 tablet_service.cc:1467] Processing CreateTablet for tablet 5cccef1ee4db4c19b913b5d1206f9fa1 (DEFAULT_TABLE table=default.Table_247c9acc11f648b281616ab9d9ea7fc2 [id=9aca53e4ba5b4e2b8dd83f36bf8371f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.046522 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:07.046695 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (table default.Table_159068abc4d842c693e5ae2323d1147d [id=15c9f7d5d3ff4fa4865b55c5892421fb]). Expected version 3 got 0
I20250109 20:53:07.046862 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 56932b16f94e4dd4bb505494e4267046 (DEFAULT_TABLE table=default.Table_ffcefcb7b4a548e085267ae98346f908 [id=504f852fb77449fd904312040d9c4622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.047184 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56932b16f94e4dd4bb505494e4267046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.050441 894 tablet_service.cc:1467] Processing CreateTablet for tablet 56932b16f94e4dd4bb505494e4267046 (DEFAULT_TABLE table=default.Table_ffcefcb7b4a548e085267ae98346f908 [id=504f852fb77449fd904312040d9c4622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.050763 883 tablet_service.cc:1467] Processing CreateTablet for tablet 20de86c42ef748368f0af66f774eecfc (DEFAULT_TABLE table=default.Table_fe13a1782fc74940871718d79b25d8b0 [id=5eec98a4754f49a693c8a101ab6e3555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.051052 885 tablet_service.cc:1467] Processing CreateTablet for tablet 4226f0393756477c82622fbe07e489b5 (DEFAULT_TABLE table=default.Table_0e4aa7de575f45cfbbd9528398ca6aaf [id=1119463f2c874e87bba18402940797a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.051334 888 tablet_service.cc:1467] Processing CreateTablet for tablet 19d96d2a15f744f0bee1ad8ab18d16cd (DEFAULT_TABLE table=default.Table_494f8c109277462ab1dc23afe0455989 [id=ec3dce082c364623a5d160c179b7da2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.051622 895 tablet_service.cc:1467] Processing CreateTablet for tablet 33fe739c48c34163af8bf5e96988b0e4 (DEFAULT_TABLE table=default.Table_181b0576d1844f09b0045ac8d5f16841 [id=0ce9af2dc9524457a7d01b397ecb7c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.051895 893 tablet_service.cc:1467] Processing CreateTablet for tablet 94afec96971241cea225c2cf9da1b9c5 (DEFAULT_TABLE table=default.Table_34d03276cc5d416a85003ea188706357 [id=97d4c5663f704bccad28497bb696f72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.052191 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5cccef1ee4db4c19b913b5d1206f9fa1 (DEFAULT_TABLE table=default.Table_247c9acc11f648b281616ab9d9ea7fc2 [id=9aca53e4ba5b4e2b8dd83f36bf8371f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.046833 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5cccef1ee4db4c19b913b5d1206f9fa1 (DEFAULT_TABLE table=default.Table_247c9acc11f648b281616ab9d9ea7fc2 [id=9aca53e4ba5b4e2b8dd83f36bf8371f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.052968 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cccef1ee4db4c19b913b5d1206f9fa1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:07.054481 989 leader_election.cc:343] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.055351 989 leader_election.cc:304] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.058261 1176 tablet_bootstrap.cc:654] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.059201 747 tablet_service.cc:1467] Processing CreateTablet for tablet 20de86c42ef748368f0af66f774eecfc (DEFAULT_TABLE table=default.Table_fe13a1782fc74940871718d79b25d8b0 [id=5eec98a4754f49a693c8a101ab6e3555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.059224 749 tablet_service.cc:1467] Processing CreateTablet for tablet 56932b16f94e4dd4bb505494e4267046 (DEFAULT_TABLE table=default.Table_ffcefcb7b4a548e085267ae98346f908 [id=504f852fb77449fd904312040d9c4622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.062166 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 19d96d2a15f744f0bee1ad8ab18d16cd (DEFAULT_TABLE table=default.Table_494f8c109277462ab1dc23afe0455989 [id=ec3dce082c364623a5d160c179b7da2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.062273 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 20de86c42ef748368f0af66f774eecfc (DEFAULT_TABLE table=default.Table_fe13a1782fc74940871718d79b25d8b0 [id=5eec98a4754f49a693c8a101ab6e3555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.062481 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d96d2a15f744f0bee1ad8ab18d16cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.062685 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4226f0393756477c82622fbe07e489b5 (DEFAULT_TABLE table=default.Table_0e4aa7de575f45cfbbd9528398ca6aaf [id=1119463f2c874e87bba18402940797a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.062988 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4226f0393756477c82622fbe07e489b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.065116 2560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8328960890b142248073cec450e957d8. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:53:07.065367 2560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [sys.catalog]: This master's current role is: FOLLOWER
I20250109 20:53:07.067869 2615 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:53:07.069002 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 33fe739c48c34163af8bf5e96988b0e4 (DEFAULT_TABLE table=default.Table_181b0576d1844f09b0045ac8d5f16841 [id=0ce9af2dc9524457a7d01b397ecb7c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.069332 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fe739c48c34163af8bf5e96988b0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.071745 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56932b16f94e4dd4bb505494e4267046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.072692 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1213) took 2763 ms. Trace:
I20250109 20:53:07.072793 1160 rpcz_store.cc:276] 0109 20:53:04.309400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.309453 (+ 53us) service_pool.cc:225] Handling call
0109 20:53:07.042356 (+2732903us) ts_tablet_manager.cc:1145] Deleting tablet bd642e5ec83a4da88eaa2eea3ffca362
0109 20:53:07.069820 (+ 27464us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.072653 (+ 2833us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2732694}
I20250109 20:53:07.072988 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 94afec96971241cea225c2cf9da1b9c5 (DEFAULT_TABLE table=default.Table_34d03276cc5d416a85003ea188706357 [id=97d4c5663f704bccad28497bb696f72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.073081 1160 tablet_replica.cc:331] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.073236 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94afec96971241cea225c2cf9da1b9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.073258 1160 raft_consensus.cc:2238] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.073388 1160 raft_consensus.cc:2267] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.074147 1160 ts_tablet_manager.cc:1905] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.076143 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4226f0393756477c82622fbe07e489b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.076877 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20de86c42ef748368f0af66f774eecfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.077764 750 tablet_service.cc:1467] Processing CreateTablet for tablet 4226f0393756477c82622fbe07e489b5 (DEFAULT_TABLE table=default.Table_0e4aa7de575f45cfbbd9528398ca6aaf [id=1119463f2c874e87bba18402940797a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.077973 757 tablet_service.cc:1467] Processing CreateTablet for tablet 33fe739c48c34163af8bf5e96988b0e4 (DEFAULT_TABLE table=default.Table_181b0576d1844f09b0045ac8d5f16841 [id=0ce9af2dc9524457a7d01b397ecb7c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.078197 741 tablet_service.cc:1467] Processing CreateTablet for tablet 19d96d2a15f744f0bee1ad8ab18d16cd (DEFAULT_TABLE table=default.Table_494f8c109277462ab1dc23afe0455989 [id=ec3dce082c364623a5d160c179b7da2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.078370 755 tablet_service.cc:1467] Processing CreateTablet for tablet 94afec96971241cea225c2cf9da1b9c5 (DEFAULT_TABLE table=default.Table_34d03276cc5d416a85003ea188706357 [id=97d4c5663f704bccad28497bb696f72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.078610 2368 raft_consensus.cc:2743] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.078898 1176 tablet_bootstrap.cc:492] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:07.079006 1176 ts_tablet_manager.cc:1397] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20250109 20:53:07.079603 2368 raft_consensus.cc:491] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.079632 1176 raft_consensus.cc:357] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.079707 2368 raft_consensus.cc:513] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.079818 1176 raft_consensus.cc:383] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.079895 1176 raft_consensus.cc:738] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.080113 1176 consensus_queue.cc:260] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.080307 2368 leader_election.cc:290] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:07.080651 1176 ts_tablet_manager.cc:1428] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:07.080948 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3008c672bc544000b24b75ae366d9fb9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.081019 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3008c672bc544000b24b75ae366d9fb9" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.081496 990 leader_election.cc:343] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:07.081617 989 leader_election.cc:343] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.081722 989 leader_election.cc:304] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.082350 2475 raft_consensus.cc:2743] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.083591 2167 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47184 (request call id 0) took 7128 ms. Trace:
I20250109 20:53:07.085911 2620 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250109 20:53:07.085973 2615 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:53:07.104041 2615 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
W20250109 20:53:07.114459 713 leader_election.cc:343] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 712acab92e0f44a3a585d7d871498aa3
I20250109 20:53:07.114799 713 leader_election.cc:304] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.113030 1176 tablet_bootstrap.cc:492] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:07.114990 2533 raft_consensus.cc:2743] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.114671 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1323) took 1830 ms. Trace:
I20250109 20:53:07.115223 2552 raft_consensus.cc:491] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.115161 1157 rpcz_store.cc:276] 0109 20:53:05.283788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.283821 (+ 33us) service_pool.cc:225] Handling call
0109 20:53:06.605727 (+1321906us) ts_tablet_manager.cc:1145] Deleting tablet 85a95a31468b431fb4304cf6680016da
0109 20:53:06.619190 (+ 13463us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:06.859896 (+240706us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.036632 (+176736us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.114656 (+ 78024us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321760}
I20250109 20:53:07.115319 2552 raft_consensus.cc:513] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.115567 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1319) took 1832 ms. Trace:
I20250109 20:53:07.115633 1157 rpcz_store.cc:276] 0109 20:53:05.282815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.283984 (+ 1169us) service_pool.cc:225] Handling call
0109 20:53:07.115432 (+1831448us) ts_tablet_manager.cc:1145] Deleting tablet 31af7fe649b94a44a1fdf71bdd5fd083
0109 20:53:07.115441 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.115551 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1831310}
I20250109 20:53:07.115929 2552 leader_election.cc:290] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.116578 1157 tablet_replica.cc:331] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:53:07.116883 854 leader_election.cc:343] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 04e46f4312844b6eb29d6718f4800f98
I20250109 20:53:07.116976 1176 tablet_bootstrap.cc:654] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.117269 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 85a95a31468b431fb4304cf6680016da (table default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]) successfully deleted
W20250109 20:53:07.117456 854 leader_election.cc:343] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 04e46f4312844b6eb29d6718f4800f98
W20250109 20:53:07.117511 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 31af7fe649b94a44a1fdf71bdd5fd083 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31af7fe649b94a44a1fdf71bdd5fd083 already in progress: creating tablet
I20250109 20:53:07.117547 854 leader_election.cc:304] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.117795 1157 raft_consensus.cc:2238] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:07.118098 1157 raft_consensus.cc:2267] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.118172 2552 raft_consensus.cc:2743] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.118366 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_59ec3dfe97bc4ffab283a5868c5d01dc" } modify_external_catalogs: true
I20250109 20:53:07.118525 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_59ec3dfe97bc4ffab283a5868c5d01dc" } modify_external_catalogs: true
I20250109 20:53:07.118564 1157 ts_tablet_manager.cc:1905] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.120617 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_591218a0ea174cecb8e1b74b3ee303f5" } new_table_name: "default.Table_2b1bd98d1ba04861ae92e6af5c6fd606" modify_external_catalogs: true
I20250109 20:53:07.120980 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f4960d18c1184100bed547ae248a1b47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.123696 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e7938ab48958476eadee6c090104823c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.129047 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4960d18c1184100bed547ae248a1b47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:07.120256 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet bd642e5ec83a4da88eaa2eea3ffca362 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd642e5ec83a4da88eaa2eea3ffca362 already in progress: creating tablet
W20250109 20:53:07.129535 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7938ab48958476eadee6c090104823c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.131456 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 0f7e3652017946d481090b9940d32e97
I20250109 20:53:07.132373 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b0dd5ca642034441a0eb5c5a653b7263
I20250109 20:53:07.133000 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dd5ca642034441a0eb5c5a653b7263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.133306 2546 raft_consensus.cc:491] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.133482 2546 raft_consensus.cc:513] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.134142 2546 leader_election.cc:290] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:07.134493 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f2d468467478415ba582be6d4a9d989b
I20250109 20:53:07.135210 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7e3652017946d481090b9940d32e97 with delete_type TABLET_DATA_DELETED (replaced by tablet 613fef2a18fd483fb58ca66c4c1b346e) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.137423 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b50c91807734e86840e874886dc0749" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.137903 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20de86c42ef748368f0af66f774eecfc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:07.137969 990 leader_election.cc:343] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.138182 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94afec96971241cea225c2cf9da1b9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.138365 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4226f0393756477c82622fbe07e489b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.138327 894 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d468467478415ba582be6d4a9d989b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.138512 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cccef1ee4db4c19b913b5d1206f9fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.140841 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fe739c48c34163af8bf5e96988b0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.143114 454 catalog_manager.cc:6596] ReplaceTablet: tablet 0f7e3652017946d481090b9940d32e97 deleted and replaced by tablet 613fef2a18fd483fb58ca66c4c1b346e
I20250109 20:53:07.144897 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56932b16f94e4dd4bb505494e4267046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.149587 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d96d2a15f744f0bee1ad8ab18d16cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.157433 894 tablet_service.cc:1467] Processing CreateTablet for tablet b1901b8928984d8eafb2b4e74e662161 (DEFAULT_TABLE table=default.Table_40f47b080f4a4134ad0921f8bcfe2f23 [id=faeab475cc474d31b8f1815f5aca6b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.157965 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7db9bca84dfb43f3aca6f12009c4007e (DEFAULT_TABLE table=default.Table_94097e767754490fa0d9c7402cc172dd [id=131b15ca7d8549d0ad4f4c848d3ec4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.158033 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3756b8ca7bd24480bf472479ebdd8425 (DEFAULT_TABLE table=default.Table_33baa454083e4b7eb40552ba4fc70124 [id=c1fa1b021b62482d8021f98bd92834e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.158540 887 tablet_service.cc:1467] Processing CreateTablet for tablet 63a665fcf6ba43f59de8b945c5534d8e (DEFAULT_TABLE table=default.Table_a017309fe2384e848ad25b4288eb890b [id=b35eec5ebf46429ab76ed31bb690a1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.159011 881 tablet_service.cc:1467] Processing CreateTablet for tablet fe7a4def93744d229a7fe4a53f62677f (DEFAULT_TABLE table=default.Table_dce041dd4c504e0694d63c2ada509f81 [id=93119db6ae7940109c2a7bd6be3caea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.159368 884 tablet_service.cc:1467] Processing CreateTablet for tablet 2223edb514244b58a18be8ca186cabb3 (DEFAULT_TABLE table=default.Table_c2360ad67a3340189132339b30692659 [id=dff850bd10984dabb87cb940dd8fb72b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.159492 1019 tablet_service.cc:1467] Processing CreateTablet for tablet b1901b8928984d8eafb2b4e74e662161 (DEFAULT_TABLE table=default.Table_40f47b080f4a4134ad0921f8bcfe2f23 [id=faeab475cc474d31b8f1815f5aca6b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.159747 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6daac072067b4071bf2280f05c47f14b (DEFAULT_TABLE table=default.Table_2e425a9a8d0f49acab65342112e3e64c [id=5823b8261bd14d89b878e84c05a40070]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.159725 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 63a665fcf6ba43f59de8b945c5534d8e (DEFAULT_TABLE table=default.Table_a017309fe2384e848ad25b4288eb890b [id=b35eec5ebf46429ab76ed31bb690a1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.160105 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 6daac072067b4071bf2280f05c47f14b (DEFAULT_TABLE table=default.Table_2e425a9a8d0f49acab65342112e3e64c [id=5823b8261bd14d89b878e84c05a40070]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.160100 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 2223edb514244b58a18be8ca186cabb3 (DEFAULT_TABLE table=default.Table_c2360ad67a3340189132339b30692659 [id=dff850bd10984dabb87cb940dd8fb72b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.160100 1032 tablet_service.cc:1467] Processing CreateTablet for tablet fe7a4def93744d229a7fe4a53f62677f (DEFAULT_TABLE table=default.Table_dce041dd4c504e0694d63c2ada509f81 [id=93119db6ae7940109c2a7bd6be3caea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.160097 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 7db9bca84dfb43f3aca6f12009c4007e (DEFAULT_TABLE table=default.Table_94097e767754490fa0d9c7402cc172dd [id=131b15ca7d8549d0ad4f4c848d3ec4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.160528 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3756b8ca7bd24480bf472479ebdd8425 (DEFAULT_TABLE table=default.Table_33baa454083e4b7eb40552ba4fc70124 [id=c1fa1b021b62482d8021f98bd92834e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.161542 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7e3652017946d481090b9940d32e97 with delete_type TABLET_DATA_DELETED (replaced by tablet 613fef2a18fd483fb58ca66c4c1b346e) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.161942 739 tablet_service.cc:1467] Processing CreateTablet for tablet 63a665fcf6ba43f59de8b945c5534d8e (DEFAULT_TABLE table=default.Table_a017309fe2384e848ad25b4288eb890b [id=b35eec5ebf46429ab76ed31bb690a1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.162113 743 tablet_service.cc:1467] Processing CreateTablet for tablet 2223edb514244b58a18be8ca186cabb3 (DEFAULT_TABLE table=default.Table_c2360ad67a3340189132339b30692659 [id=dff850bd10984dabb87cb940dd8fb72b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.162297 742 tablet_service.cc:1467] Processing CreateTablet for tablet b1901b8928984d8eafb2b4e74e662161 (DEFAULT_TABLE table=default.Table_40f47b080f4a4134ad0921f8bcfe2f23 [id=faeab475cc474d31b8f1815f5aca6b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.162369 744 tablet_service.cc:1467] Processing CreateTablet for tablet fe7a4def93744d229a7fe4a53f62677f (DEFAULT_TABLE table=default.Table_dce041dd4c504e0694d63c2ada509f81 [id=93119db6ae7940109c2a7bd6be3caea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.161438 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dd5ca642034441a0eb5c5a653b7263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.162994 745 tablet_service.cc:1467] Processing CreateTablet for tablet 7db9bca84dfb43f3aca6f12009c4007e (DEFAULT_TABLE table=default.Table_94097e767754490fa0d9c7402cc172dd [id=131b15ca7d8549d0ad4f4c848d3ec4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.163462 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6daac072067b4071bf2280f05c47f14b (DEFAULT_TABLE table=default.Table_2e425a9a8d0f49acab65342112e3e64c [id=5823b8261bd14d89b878e84c05a40070]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.162706 740 tablet_service.cc:1467] Processing CreateTablet for tablet 3756b8ca7bd24480bf472479ebdd8425 (DEFAULT_TABLE table=default.Table_33baa454083e4b7eb40552ba4fc70124 [id=c1fa1b021b62482d8021f98bd92834e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:07.164508 1176 tablet_bootstrap.cc:492] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:07.164634 1176 ts_tablet_manager.cc:1397] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20250109 20:53:07.169533 1176 raft_consensus.cc:357] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.169801 1176 raft_consensus.cc:383] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.173033 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2223edb514244b58a18be8ca186cabb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.175102 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6daac072067b4071bf2280f05c47f14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.175591 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db9bca84dfb43f3aca6f12009c4007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.176017 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7a4def93744d229a7fe4a53f62677f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.176599 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a665fcf6ba43f59de8b945c5534d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.176791 2546 raft_consensus.cc:491] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.176997 2546 raft_consensus.cc:513] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.177220 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d468467478415ba582be6d4a9d989b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.177927 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1901b8928984d8eafb2b4e74e662161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.178092 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3293f49190684b72a5c41031e621bd36" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.178442 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3756b8ca7bd24480bf472479ebdd8425. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:07.178478 712 consensus_peers.cc:487] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250109 20:53:07.178617 990 leader_election.cc:343] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.178669 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f7e3652017946d481090b9940d32e97 with delete_type TABLET_DATA_DELETED (replaced by tablet 613fef2a18fd483fb58ca66c4c1b346e) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.179024 2546 leader_election.cc:290] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:07.179695 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3293f49190684b72a5c41031e621bd36" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.180217 989 leader_election.cc:343] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.180317 989 leader_election.cc:304] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.180485 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d468467478415ba582be6d4a9d989b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.183655 2546 raft_consensus.cc:2743] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.183718 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2223edb514244b58a18be8ca186cabb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.185633 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dd5ca642034441a0eb5c5a653b7263 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.185964 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b50c91807734e86840e874886dc0749" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.186452 989 leader_election.cc:343] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.186563 989 leader_election.cc:304] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.186604 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fe739c48c34163af8bf5e96988b0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.187095 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94afec96971241cea225c2cf9da1b9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.187269 2546 raft_consensus.cc:2743] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.187537 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cccef1ee4db4c19b913b5d1206f9fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.187955 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20de86c42ef748368f0af66f774eecfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.188349 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d96d2a15f744f0bee1ad8ab18d16cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.188822 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1901b8928984d8eafb2b4e74e662161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.189229 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db9bca84dfb43f3aca6f12009c4007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.189958 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6daac072067b4071bf2280f05c47f14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.190333 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7a4def93744d229a7fe4a53f62677f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.190683 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a665fcf6ba43f59de8b945c5534d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.190822 2145 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "44d473955919435cb679edd5f8afe2c2" instance_seqno: 1736455961526528) as {username='slave'} at 127.0.104.2:55611; Asking this server to re-register.
I20250109 20:53:07.191234 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3756b8ca7bd24480bf472479ebdd8425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.192860 2625 tablet.cc:1721] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:07.201846 1166 ts_tablet_manager.cc:1918] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.201977 1166 log.cc:1198] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4870a667cd3e410590923c9e2497af54
I20250109 20:53:07.202330 1166 ts_tablet_manager.cc:1939] T 4870a667cd3e410590923c9e2497af54 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.203289 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1232) took 3057 ms. Trace:
I20250109 20:53:07.203377 1166 rpcz_store.cc:276] 0109 20:53:04.145758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.145972 (+ 214us) service_pool.cc:225] Handling call
0109 20:53:06.994818 (+2848846us) ts_tablet_manager.cc:1145] Deleting tablet 4870a667cd3e410590923c9e2497af54
0109 20:53:06.994827 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.172584 (+177757us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.201716 (+ 29132us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.203263 (+ 1547us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2848708}
I20250109 20:53:07.203811 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1233) took 3053 ms. Trace:
I20250109 20:53:07.203894 1166 rpcz_store.cc:276] 0109 20:53:04.150661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.150759 (+ 98us) service_pool.cc:225] Handling call
0109 20:53:07.203668 (+3052909us) ts_tablet_manager.cc:1145] Deleting tablet 7357b4f67dc6448dbb8f77dce81731f3
0109 20:53:07.203681 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.203793 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3052719}
I20250109 20:53:07.204231 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1235) took 3051 ms. Trace:
I20250109 20:53:07.204205 2145 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "09833ea46b7141d58c3899dace69e49f" instance_seqno: 1736455961029491) as {username='slave'} at 127.0.104.1:33381; Asking this server to re-register.
I20250109 20:53:07.204300 1166 rpcz_store.cc:276] 0109 20:53:04.152685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.152940 (+ 255us) service_pool.cc:225] Handling call
0109 20:53:07.204131 (+3051191us) ts_tablet_manager.cc:1145] Deleting tablet cb880d4da1ca4c2c851fc8c1e0846ce4
0109 20:53:07.204143 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.204216 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3051021}
I20250109 20:53:07.204542 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4870a667cd3e410590923c9e2497af54 (table default.Table_03ed60a072e941b6b5b9a8a477aa9020 [id=1e9c11e6b9294f56bb2efd10bef9af20]) successfully deleted
I20250109 20:53:07.204645 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1237) took 3041 ms. Trace:
I20250109 20:53:07.205005 2552 raft_consensus.cc:491] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.205039 1157 ts_tablet_manager.cc:1918] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20250109 20:53:07.205053 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7357b4f67dc6448dbb8f77dce81731f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7357b4f67dc6448dbb8f77dce81731f3 already in progress: creating tablet
I20250109 20:53:07.205148 1157 log.cc:1198] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/b4b31bd4072941d5aaf8c43dada3ccc7
I20250109 20:53:07.205154 2552 raft_consensus.cc:513] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.205264 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet cb880d4da1ca4c2c851fc8c1e0846ce4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb880d4da1ca4c2c851fc8c1e0846ce4 already in progress: creating tablet
I20250109 20:53:07.205516 1157 ts_tablet_manager.cc:1939] T b4b31bd4072941d5aaf8c43dada3ccc7 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.205837 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a665fcf6ba43f59de8b945c5534d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.205813 2552 leader_election.cc:290] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.206359 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2223edb514244b58a18be8ca186cabb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.206859 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1901b8928984d8eafb2b4e74e662161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.207286 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7a4def93744d229a7fe4a53f62677f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:07.207302 854 leader_election.cc:343] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: e16a1d9a5b374b8dbf725eb32ffadb9e
W20250109 20:53:07.207652 854 leader_election.cc:343] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: e16a1d9a5b374b8dbf725eb32ffadb9e
I20250109 20:53:07.207742 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db9bca84dfb43f3aca6f12009c4007e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.207746 854 leader_election.cc:304] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.208204 1166 rpcz_store.cc:276] 0109 20:53:04.163071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.163115 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:07.204542 (+3041427us) ts_tablet_manager.cc:1145] Deleting tablet d79437d800f44c5a8222eea7dc2cd4e3
0109 20:53:07.204551 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.204631 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3041287}
I20250109 20:53:07.208510 1166 tablet_replica.cc:331] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.208510 2552 raft_consensus.cc:2743] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:07.208629 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d79437d800f44c5a8222eea7dc2cd4e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d79437d800f44c5a8222eea7dc2cd4e3 already in progress: creating tablet
I20250109 20:53:07.208765 1166 raft_consensus.cc:2238] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.208810 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6daac072067b4071bf2280f05c47f14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.208945 1166 raft_consensus.cc:2267] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.209230 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3756b8ca7bd24480bf472479ebdd8425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:07.209498 1166 ts_tablet_manager.cc:1905] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.169873 1176 raft_consensus.cc:738] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.210397 1176 consensus_queue.cc:260] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.210948 1176 ts_tablet_manager.cc:1428] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20250109 20:53:07.211231 1176 tablet_bootstrap.cc:492] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:07.213961 964 heartbeater.cc:463] Registering TS with master...
I20250109 20:53:07.221760 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_34d03276cc5d416a85003ea188706357" } modify_external_catalogs: true
I20250109 20:53:07.221968 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_34d03276cc5d416a85003ea188706357" } modify_external_catalogs: true
I20250109 20:53:07.223811 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_09ed969c8f9a442392025cc69ce2ea70" } new_table_name: "default.Table_42a10c51a1f649789d65f93ad913becc" modify_external_catalogs: true
I20250109 20:53:07.223987 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1324) took 1933 ms. Trace:
I20250109 20:53:07.224073 1157 rpcz_store.cc:276] 0109 20:53:05.290864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.291497 (+ 633us) service_pool.cc:225] Handling call
0109 20:53:07.115831 (+1824334us) ts_tablet_manager.cc:1145] Deleting tablet b4b31bd4072941d5aaf8c43dada3ccc7
0109 20:53:07.115839 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.155424 (+ 39585us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.204922 (+ 49498us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.223967 (+ 19045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1824133}
I20250109 20:53:07.224426 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1329) took 1932 ms. Trace:
I20250109 20:53:07.224496 1157 rpcz_store.cc:276] 0109 20:53:05.292171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.292211 (+ 40us) service_pool.cc:225] Handling call
0109 20:53:07.224315 (+1932104us) ts_tablet_manager.cc:1145] Deleting tablet 5e843079de154ee5b1b340b2d0d11c7f
0109 20:53:07.224324 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.224408 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927527}
I20250109 20:53:07.224757 1157 tablet_replica.cc:331] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.224925 1157 raft_consensus.cc:2238] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.225052 1157 raft_consensus.cc:2267] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.225523 1157 ts_tablet_manager.cc:1905] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.229060 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f9795a157f5045f4bd26f3db299b99ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.229612 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9795a157f5045f4bd26f3db299b99ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.229948 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_8ca6e16543a94e608f9e3a73747ec6d2 [id=2ab3ba8731124704889904608a2f0cd3]) successfully deleted
W20250109 20:53:07.230142 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5e843079de154ee5b1b340b2d0d11c7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e843079de154ee5b1b340b2d0d11c7f already in progress: creating tablet
I20250109 20:53:07.237700 455 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 2420b285905342b3820f9549c8f0c4ad (table default.Table_bb47b9b8617f4706aeb895a16ded22ee [id=ec17429d6df548118b1e53e77bac80c0]). Expected version 2 got 0
I20250109 20:53:07.237900 1157 ts_tablet_manager.cc:1918] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.237995 1157 log.cc:1198] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/82209a6b39bb48e791ff63a8167c9765
I20250109 20:53:07.238222 1158 tablet_bootstrap.cc:492] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:07.238269 1157 ts_tablet_manager.cc:1939] T 82209a6b39bb48e791ff63a8167c9765 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.238955 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1334) took 1930 ms. Trace:
I20250109 20:53:07.239028 1157 rpcz_store.cc:276] 0109 20:53:05.308228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.308270 (+ 42us) service_pool.cc:225] Handling call
0109 20:53:07.224685 (+1916415us) ts_tablet_manager.cc:1145] Deleting tablet 82209a6b39bb48e791ff63a8167c9765
0109 20:53:07.224694 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.228145 (+ 3451us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.237820 (+ 9675us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.238938 (+ 1118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1916281}
I20250109 20:53:07.239334 1157 tablet_replica.cc:331] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.239445 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 82209a6b39bb48e791ff63a8167c9765 (table default.Table_06b47da1327e425989c115839e9aecbe [id=cced112727b0498f955953f2b2d29551]) successfully deleted
I20250109 20:53:07.239500 1157 raft_consensus.cc:2238] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.239619 1157 raft_consensus.cc:2267] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.240053 1157 ts_tablet_manager.cc:1905] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.242034 1158 tablet_bootstrap.cc:654] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.244701 1160 ts_tablet_manager.cc:1918] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.244804 1160 log.cc:1198] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/9558dfe563d94482bf496e19b8a80edc
I20250109 20:53:07.245059 1160 ts_tablet_manager.cc:1939] T 9558dfe563d94482bf496e19b8a80edc P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.251381 1157 ts_tablet_manager.cc:1918] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.251483 1157 log.cc:1198] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/0cede2a1882946a181b18d66f12c5114
I20250109 20:53:07.251722 1157 ts_tablet_manager.cc:1939] T 0cede2a1882946a181b18d66f12c5114 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.252379 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1338) took 1942 ms. Trace:
I20250109 20:53:07.252449 1157 rpcz_store.cc:276] 0109 20:53:05.309779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.309826 (+ 47us) service_pool.cc:225] Handling call
0109 20:53:07.239275 (+1929449us) ts_tablet_manager.cc:1145] Deleting tablet 0cede2a1882946a181b18d66f12c5114
0109 20:53:07.239285 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.248921 (+ 9636us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.251324 (+ 2403us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.252362 (+ 1038us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929279}
I20250109 20:53:07.252822 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 0cede2a1882946a181b18d66f12c5114 (table default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]) successfully deleted
I20250109 20:53:07.253894 2475 raft_consensus.cc:491] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.254022 2475 raft_consensus.cc:513] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.254690 2475 leader_election.cc:290] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:07.254824 2475 raft_consensus.cc:491] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.254935 2475 raft_consensus.cc:513] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.255070 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af9563f6efd47059caac7c86c41d43f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.255486 2475 leader_election.cc:290] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:53:07.255692 990 leader_election.cc:343] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.256325 1165 tablet_bootstrap.cc:492] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.256435 1165 ts_tablet_manager.cc:1397] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.214s user 0.000s sys 0.007s
I20250109 20:53:07.256778 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b285905342b3820f9549c8f0c4ad" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:07.257086 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af9563f6efd47059caac7c86c41d43f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:07.257699 1165 raft_consensus.cc:357] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.257861 1165 raft_consensus.cc:383] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.257901 1165 raft_consensus.cc:738] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:53:07.258031 989 leader_election.cc:343] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.258060 1165 consensus_queue.cc:260] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.258217 989 leader_election.cc:343] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.258316 989 leader_election.cc:304] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.258744 2546 raft_consensus.cc:2743] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.258862 1165 ts_tablet_manager.cc:1428] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:07.259080 1165 tablet_bootstrap.cc:492] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.260288 1165 tablet_bootstrap.cc:654] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.262360 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f] alter complete (version 4)
I20250109 20:53:07.263309 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_190f7022156a45b0b2905f93a7f274df" } new_table_name: "default.Table_a2bd3c2b8a824efa9bba28ac2162fb61" modify_external_catalogs: true
I20250109 20:53:07.263657 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b285905342b3820f9549c8f0c4ad" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.263751 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1218) took 2920 ms. Trace:
I20250109 20:53:07.263813 1160 rpcz_store.cc:276] 0109 20:53:04.343565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.343618 (+ 53us) service_pool.cc:225] Handling call
0109 20:53:07.073041 (+2729423us) ts_tablet_manager.cc:1145] Deleting tablet 9558dfe563d94482bf496e19b8a80edc
0109 20:53:07.073049 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.181114 (+108065us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.244624 (+ 63510us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.263737 (+ 19113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2673469}
I20250109 20:53:07.264163 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 9558dfe563d94482bf496e19b8a80edc (table default.Table_ec9106622eb347769d374f0583168a54 [id=095b22eda1d34de4b3394189471c53c6]) successfully deleted
W20250109 20:53:07.264842 990 leader_election.cc:343] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.264935 990 leader_election.cc:304] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.265182 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1224) took 2812 ms. Trace:
I20250109 20:53:07.265249 1160 rpcz_store.cc:276] 0109 20:53:04.453050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.453135 (+ 85us) service_pool.cc:225] Handling call
0109 20:53:07.264386 (+2811251us) ts_tablet_manager.cc:1145] Deleting tablet 801838c21f70424ba1b2d89585bda643
0109 20:53:07.264394 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.265170 (+ 776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2811040}
W20250109 20:53:07.265594 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 801838c21f70424ba1b2d89585bda643 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 801838c21f70424ba1b2d89585bda643 already in progress: creating tablet
I20250109 20:53:07.265738 2475 raft_consensus.cc:2743] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.266054 1160 tablet_replica.cc:331] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.266222 1160 raft_consensus.cc:2238] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.266356 1160 raft_consensus.cc:2267] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.266806 1160 ts_tablet_manager.cc:1905] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.271862 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 006475691733470f8a0468365972dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.276759 915 raft_consensus.cc:3054] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:07.285221 1158 tablet_bootstrap.cc:492] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:07.285394 1158 ts_tablet_manager.cc:1397] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.001s
I20250109 20:53:07.286013 1166 ts_tablet_manager.cc:1918] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.286123 1166 log.cc:1198] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/0019a01afbcc452a9a9374c92dc131f3
I20250109 20:53:07.286180 1158 raft_consensus.cc:357] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.286378 1166 ts_tablet_manager.cc:1939] T 0019a01afbcc452a9a9374c92dc131f3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.286394 1158 raft_consensus.cc:383] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.286494 1158 raft_consensus.cc:738] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.286767 1158 consensus_queue.cc:260] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.287021 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1230) took 3141 ms. Trace:
I20250109 20:53:07.287109 1166 rpcz_store.cc:276] 0109 20:53:04.145561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.172913 (+ 27352us) service_pool.cc:225] Handling call
0109 20:53:07.208463 (+3035550us) ts_tablet_manager.cc:1145] Deleting tablet 0019a01afbcc452a9a9374c92dc131f3
0109 20:53:07.208475 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.283271 (+ 74796us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.285944 (+ 2673us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.287004 (+ 1060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3035364}
I20250109 20:53:07.287535 1158 ts_tablet_manager.cc:1428] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:53:07.287705 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1239) took 3113 ms. Trace:
I20250109 20:53:07.287786 1166 rpcz_store.cc:276] 0109 20:53:04.174458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.180860 (+ 6402us) service_pool.cc:225] Handling call
0109 20:53:07.287595 (+3106735us) ts_tablet_manager.cc:1145] Deleting tablet ac867ce748f749d88885ef13654983ec
0109 20:53:07.287605 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.287691 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3106195}
I20250109 20:53:07.287859 1158 tablet_bootstrap.cc:492] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:07.288133 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ac867ce748f749d88885ef13654983ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac867ce748f749d88885ef13654983ec already in progress: creating tablet
I20250109 20:53:07.288488 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1251) took 2978 ms. Trace:
I20250109 20:53:07.288569 1166 rpcz_store.cc:276] 0109 20:53:04.309799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.309871 (+ 72us) service_pool.cc:225] Handling call
0109 20:53:07.288400 (+2978529us) ts_tablet_manager.cc:1145] Deleting tablet bd642e5ec83a4da88eaa2eea3ffca362
0109 20:53:07.288408 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.288475 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2978349}
W20250109 20:53:07.288990 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet bd642e5ec83a4da88eaa2eea3ffca362 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd642e5ec83a4da88eaa2eea3ffca362 already in progress: creating tablet
I20250109 20:53:07.289263 1166 tablet_replica.cc:331] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.289412 1158 tablet_bootstrap.cc:654] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.289655 1166 raft_consensus.cc:2238] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.289796 1166 raft_consensus.cc:2267] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.290058 823 heartbeater.cc:463] Registering TS with master...
I20250109 20:53:07.290295 1166 ts_tablet_manager.cc:1905] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.293371 1160 ts_tablet_manager.cc:1918] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.294543 1157 tablet_replica.cc:331] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.294730 1157 raft_consensus.cc:2238] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.294845 1160 log.cc:1198] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ffd3a1b1d3e44feda6dbaa8556e377fc
I20250109 20:53:07.295148 1160 ts_tablet_manager.cc:1939] T ffd3a1b1d3e44feda6dbaa8556e377fc P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.295298 1157 raft_consensus.cc:2267] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.295544 1166 ts_tablet_manager.cc:1918] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.295779 2533 raft_consensus.cc:491] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.295888 2533 raft_consensus.cc:513] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.296139 1157 ts_tablet_manager.cc:1905] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.296492 2533 leader_election.cc:290] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:07.297338 713 leader_election.cc:343] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 006475691733470f8a0468365972dfd2
I20250109 20:53:07.294371 1165 tablet_bootstrap.cc:492] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.297046 2552 raft_consensus.cc:491] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.300310 1165 ts_tablet_manager.cc:1397] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.001s
I20250109 20:53:07.300482 1166 log.cc:1198] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/dc235614171f4a61b75492ddeb3f0fe6
I20250109 20:53:07.300379 2552 raft_consensus.cc:513] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.300770 1166 ts_tablet_manager.cc:1939] T dc235614171f4a61b75492ddeb3f0fe6 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.301048 1165 raft_consensus.cc:357] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.301236 1165 raft_consensus.cc:383] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.301287 1165 raft_consensus.cc:738] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.297075 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006475691733470f8a0468365972dfd2" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:07.301458 1165 consensus_queue.cc:260] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.301903 854 leader_election.cc:343] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 14fb1017fe6c4f5abd640ad1f6214232
W20250109 20:53:07.302161 712 leader_election.cc:343] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.302254 712 leader_election.cc:304] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:07.302306 854 leader_election.cc:343] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 14fb1017fe6c4f5abd640ad1f6214232
I20250109 20:53:07.302397 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1225) took 2815 ms. Trace:
I20250109 20:53:07.302436 854 leader_election.cc:304] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.302460 1160 rpcz_store.cc:276] 0109 20:53:04.487439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.487491 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:07.266010 (+2778519us) ts_tablet_manager.cc:1145] Deleting tablet ffd3a1b1d3e44feda6dbaa8556e377fc
0109 20:53:07.266018 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.269653 (+ 3635us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.293252 (+ 23599us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.302381 (+ 9129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2778303}
I20250109 20:53:07.302587 2533 raft_consensus.cc:2743] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.302949 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ffd3a1b1d3e44feda6dbaa8556e377fc (table default.Table_b856f34058994714afb4ce41d44f5384 [id=7adda20a8381463a9f6decd4c7cd045d]) successfully deleted
I20250109 20:53:07.303395 2632 raft_consensus.cc:2743] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.315168 1157 ts_tablet_manager.cc:1918] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.319273 1157 log.cc:1198] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/5f953f43877c4a738ffd8255afbf8275
I20250109 20:53:07.319644 1157 ts_tablet_manager.cc:1939] T 5f953f43877c4a738ffd8255afbf8275 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.320361 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1339) took 2010 ms. Trace:
I20250109 20:53:07.308903 1165 ts_tablet_manager.cc:1428] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250109 20:53:07.321182 1165 tablet_bootstrap.cc:492] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.309005 1160 tablet_replica.cc:331] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.301049 2552 leader_election.cc:290] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.321820 1160 raft_consensus.cc:2238] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.321731 1157 rpcz_store.cc:276] 0109 20:53:05.310173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.310210 (+ 37us) service_pool.cc:225] Handling call
0109 20:53:07.294495 (+1984285us) ts_tablet_manager.cc:1145] Deleting tablet 5f953f43877c4a738ffd8255afbf8275
0109 20:53:07.294506 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.299222 (+ 4716us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.315075 (+ 15853us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.320342 (+ 5267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1984168}
I20250109 20:53:07.321969 1160 raft_consensus.cc:2267] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.322463 1160 ts_tablet_manager.cc:1905] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.322685 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 5f953f43877c4a738ffd8255afbf8275 (table default.Table_3254b432932b4d9bb83cb8a2f26e5514 [id=fa8fa63dbd44442b81a3b9640d42680e]) successfully deleted
I20250109 20:53:07.323083 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1340) took 2012 ms. Trace:
I20250109 20:53:07.323299 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 94afec96971241cea225c2cf9da1b9c5
I20250109 20:53:07.323930 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 94afec96971241cea225c2cf9da1b9c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.324471 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 51b760fcb4fc466ba49d456892d43f3c
W20250109 20:53:07.324849 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e15c4d4aac9445bc9e85fc7693ebdec8 consensus state has no leader
I20250109 20:53:07.323426 1157 rpcz_store.cc:276] 0109 20:53:05.310519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.310558 (+ 39us) service_pool.cc:225] Handling call
0109 20:53:07.322967 (+2012409us) ts_tablet_manager.cc:1145] Deleting tablet b54b66ca10114330b3485f67eb076064
0109 20:53:07.322977 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.323068 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2012282}
I20250109 20:53:07.326117 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_494f8c109277462ab1dc23afe0455989" } new_table_name: "default.Table_faa0b66a512f4ed3a52cbfcc779d81b1" modify_external_catalogs: true
I20250109 20:53:07.326175 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ebeffe516c4c239ae2ee9040276e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.328509 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 086c6b621cae4214bcec4daa1a725272
W20250109 20:53:07.328537 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b54b66ca10114330b3485f67eb076064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54b66ca10114330b3485f67eb076064 already in progress: creating tablet
I20250109 20:53:07.329370 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_27aa42c2c837429bbca3a60650ee51ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.329512 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 086c6b621cae4214bcec4daa1a725272 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:07.330008 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27aa42c2c837429bbca3a60650ee51ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.330591 1157 tablet_replica.cc:331] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.330762 1157 raft_consensus.cc:2238] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.330889 1157 raft_consensus.cc:2267] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.331313 1157 ts_tablet_manager.cc:1905] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.331589 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_80bd0d8dd57e4f28a7edde0d3e630867"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.332055 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80bd0d8dd57e4f28a7edde0d3e630867 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.332932 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 94afec96971241cea225c2cf9da1b9c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.333618 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 086c6b621cae4214bcec4daa1a725272 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.333859 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 086c6b621cae4214bcec4daa1a725272 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.334216 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1259) took 2970 ms. Trace:
I20250109 20:53:07.334296 1166 rpcz_store.cc:276] 0109 20:53:04.383286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.383339 (+ 53us) service_pool.cc:225] Handling call
0109 20:53:07.289225 (+2905886us) ts_tablet_manager.cc:1145] Deleting tablet dc235614171f4a61b75492ddeb3f0fe6
0109 20:53:07.289233 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.292759 (+ 3526us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.295480 (+ 2721us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.334197 (+ 38717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2885403}
I20250109 20:53:07.334623 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet dc235614171f4a61b75492ddeb3f0fe6 (table default.Table_fc79e5e4f5114b189904f056b147d163 [id=aea1bb0e692345379ba28d2c4c7f3fb8]) successfully deleted
I20250109 20:53:07.334650 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 94afec96971241cea225c2cf9da1b9c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.337587 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7eaae83ca44c4bdcae00fed3b9e2a6d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.338110 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7eaae83ca44c4bdcae00fed3b9e2a6d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.339040 1160 ts_tablet_manager.cc:1918] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.339140 1160 log.cc:1198] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/93d409409813420689134773c627833b
I20250109 20:53:07.339421 1160 ts_tablet_manager.cc:1939] T 93d409409813420689134773c627833b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.339681 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1227) took 2836 ms. Trace:
I20250109 20:53:07.339764 1160 rpcz_store.cc:276] 0109 20:53:04.502890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.502936 (+ 46us) service_pool.cc:225] Handling call
0109 20:53:07.308502 (+2805566us) ts_tablet_manager.cc:1145] Deleting tablet 93d409409813420689134773c627833b
0109 20:53:07.308966 (+ 464us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.331553 (+ 22587us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.338953 (+ 7400us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.339663 (+ 710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2805311}
I20250109 20:53:07.340210 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1226) took 2837 ms. Trace:
I20250109 20:53:07.340287 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1263) took 2751 ms. Trace:
I20250109 20:53:07.340278 1160 rpcz_store.cc:276] 0109 20:53:04.502753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.502814 (+ 61us) service_pool.cc:225] Handling call
0109 20:53:07.340104 (+2837290us) ts_tablet_manager.cc:1145] Deleting tablet d0dc4addb7fd4d23822b25af5e288118
0109 20:53:07.340111 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.340198 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2836806}
I20250109 20:53:07.340327 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 93d409409813420689134773c627833b (table default.Table_1ee95fba580446b8926cbc779fca720f [id=a4a7c86d0df2412a83f12157c967aae4]) successfully deleted
I20250109 20:53:07.340350 1166 rpcz_store.cc:276] 0109 20:53:04.588580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.588629 (+ 49us) service_pool.cc:225] Handling call
0109 20:53:07.340173 (+2751544us) ts_tablet_manager.cc:1145] Deleting tablet d0dc4addb7fd4d23822b25af5e288118
0109 20:53:07.340183 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.340270 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2734686}
I20250109 20:53:07.340582 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1228) took 2813 ms. Trace:
I20250109 20:53:07.340670 1160 rpcz_store.cc:276] 0109 20:53:04.527585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.527639 (+ 54us) service_pool.cc:225] Handling call
0109 20:53:07.340488 (+2812849us) ts_tablet_manager.cc:1145] Deleting tablet 75ad55b91a6c405aae3ed456a8c4d9a3
0109 20:53:07.340496 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.340568 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2812678}
I20250109 20:53:07.341012 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b760fcb4fc466ba49d456892d43f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:07.341041 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d0dc4addb7fd4d23822b25af5e288118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0dc4addb7fd4d23822b25af5e288118 already in progress: creating tablet
W20250109 20:53:07.341277 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ad55b91a6c405aae3ed456a8c4d9a3 already in progress: creating tablet
W20250109 20:53:07.342082 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d0dc4addb7fd4d23822b25af5e288118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0dc4addb7fd4d23822b25af5e288118 already in progress: creating tablet
I20250109 20:53:07.342582 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b760fcb4fc466ba49d456892d43f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.343535 1176 tablet_bootstrap.cc:654] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.341064 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1229) took 2811 ms. Trace:
I20250109 20:53:07.345021 1160 rpcz_store.cc:276] 0109 20:53:04.529758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.529805 (+ 47us) service_pool.cc:225] Handling call
0109 20:53:07.340962 (+2811157us) ts_tablet_manager.cc:1145] Deleting tablet b087b7546a39451089666a36d029e645
0109 20:53:07.340974 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.341053 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":149,"tablet-delete.queue_time_us":2810975}
I20250109 20:53:07.345348 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b760fcb4fc466ba49d456892d43f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.346444 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1266) took 2743 ms. Trace:
I20250109 20:53:07.346529 1166 rpcz_store.cc:276] 0109 20:53:04.602613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.602671 (+ 58us) service_pool.cc:225] Handling call
0109 20:53:07.341118 (+2738447us) ts_tablet_manager.cc:1145] Deleting tablet c6bc95b32aa04e25bbde02bf5e8fe4d9
0109 20:53:07.341127 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.341197 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2734238}
I20250109 20:53:07.347801 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1264) took 2745 ms. Trace:
I20250109 20:53:07.347874 1166 rpcz_store.cc:276] 0109 20:53:04.602465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.604620 (+ 2155us) service_pool.cc:225] Handling call
0109 20:53:07.347699 (+2743079us) ts_tablet_manager.cc:1145] Deleting tablet 75ad55b91a6c405aae3ed456a8c4d9a3
0109 20:53:07.347708 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.347786 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2740567}
W20250109 20:53:07.347987 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b087b7546a39451089666a36d029e645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b087b7546a39451089666a36d029e645 already in progress: creating tablet
I20250109 20:53:07.348162 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1265) took 2745 ms. Trace:
I20250109 20:53:07.348222 1166 rpcz_store.cc:276] 0109 20:53:04.602551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.602682 (+ 131us) service_pool.cc:225] Handling call
0109 20:53:07.348067 (+2745385us) ts_tablet_manager.cc:1145] Deleting tablet b087b7546a39451089666a36d029e645
0109 20:53:07.348074 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.348150 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2740412}
I20250109 20:53:07.348480 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1262) took 2895 ms. Trace:
I20250109 20:53:07.348544 1166 rpcz_store.cc:276] 0109 20:53:04.453241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.570983 (+117742us) service_pool.cc:225] Handling call
0109 20:53:07.348400 (+2777417us) ts_tablet_manager.cc:1145] Deleting tablet 801838c21f70424ba1b2d89585bda643
0109 20:53:07.348408 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.348469 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2731862}
W20250109 20:53:07.349009 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 already in progress: creating tablet
W20250109 20:53:07.349226 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 75ad55b91a6c405aae3ed456a8c4d9a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ad55b91a6c405aae3ed456a8c4d9a3 already in progress: creating tablet
W20250109 20:53:07.349380 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b087b7546a39451089666a36d029e645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b087b7546a39451089666a36d029e645 already in progress: creating tablet
W20250109 20:53:07.349562 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 801838c21f70424ba1b2d89585bda643 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 801838c21f70424ba1b2d89585bda643 already in progress: creating tablet
I20250109 20:53:07.352293 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1230) took 2815 ms. Trace:
I20250109 20:53:07.352396 1160 rpcz_store.cc:276] 0109 20:53:04.536472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.536532 (+ 60us) service_pool.cc:225] Handling call
0109 20:53:07.352165 (+2815633us) ts_tablet_manager.cc:1145] Deleting tablet c6bc95b32aa04e25bbde02bf5e8fe4d9
0109 20:53:07.352175 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.352271 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2815381}
I20250109 20:53:07.352769 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1245) took 2569 ms. Trace:
I20250109 20:53:07.352829 1160 rpcz_store.cc:276] 0109 20:53:04.783208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.837219 (+ 54011us) service_pool.cc:225] Handling call
0109 20:53:07.352656 (+2515437us) ts_tablet_manager.cc:1145] Deleting tablet 568e4a83fddd45c8b61849a369074c4d
0109 20:53:07.352666 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.352757 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2515167}
I20250109 20:53:07.353107 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1249) took 2464 ms. Trace:
I20250109 20:53:07.353181 1160 rpcz_store.cc:276] 0109 20:53:04.888616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.907126 (+ 18510us) service_pool.cc:225] Handling call
0109 20:53:07.353026 (+2445900us) ts_tablet_manager.cc:1145] Deleting tablet aba42482feab4d0f85fb46b4c8b01d19
0109 20:53:07.353033 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.353094 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2445702}
I20250109 20:53:07.353477 1166 tablet_replica.cc:331] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.353490 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1255) took 2408 ms. Trace:
I20250109 20:53:07.353540 1160 rpcz_store.cc:276] 0109 20:53:04.945034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.945289 (+ 255us) service_pool.cc:225] Handling call
0109 20:53:07.353411 (+2408122us) ts_tablet_manager.cc:1145] Deleting tablet 31af7fe649b94a44a1fdf71bdd5fd083
0109 20:53:07.353418 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.353478 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2407884}
I20250109 20:53:07.353652 1166 raft_consensus.cc:2238] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.353786 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1256) took 2363 ms. Trace:
I20250109 20:53:07.353796 1166 raft_consensus.cc:2267] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.353842 1160 rpcz_store.cc:276] 0109 20:53:04.990411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.990617 (+ 206us) service_pool.cc:225] Handling call
0109 20:53:07.353707 (+2363090us) ts_tablet_manager.cc:1145] Deleting tablet ec1eaa6bc5a84b69840c540319265efb
0109 20:53:07.353714 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.353773 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2362876}
I20250109 20:53:07.354133 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1257) took 2363 ms. Trace:
I20250109 20:53:07.354194 1160 rpcz_store.cc:276] 0109 20:53:04.990501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.990717 (+ 216us) service_pool.cc:225] Handling call
0109 20:53:07.354049 (+2363332us) ts_tablet_manager.cc:1145] Deleting tablet 5f05e17ca65243a898a7c75d3f453706
0109 20:53:07.354056 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.354119 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31,"tablet-delete.queue_time_us":2363208}
I20250109 20:53:07.354324 1166 ts_tablet_manager.cc:1905] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.354503 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1258) took 2363 ms. Trace:
I20250109 20:53:07.354557 1160 rpcz_store.cc:276] 0109 20:53:04.990573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.990617 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:07.354406 (+2363789us) ts_tablet_manager.cc:1145] Deleting tablet 908781d94a054386b7d7c06c2c9fa3ca
0109 20:53:07.354414 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.354490 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2363418}
I20250109 20:53:07.354792 1160 tablet_replica.cc:331] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.354948 1160 raft_consensus.cc:2238] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.355062 1160 raft_consensus.cc:2267] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.355487 1160 ts_tablet_manager.cc:1905] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.359176 1157 ts_tablet_manager.cc:1918] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.359264 1157 log.cc:1198] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/724d15e05a864852a118badae13a5d3d
I20250109 20:53:07.359503 1157 ts_tablet_manager.cc:1939] T 724d15e05a864852a118badae13a5d3d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.360270 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1341) took 2049 ms. Trace:
I20250109 20:53:07.360344 1157 rpcz_store.cc:276] 0109 20:53:05.310925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.310977 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:07.327883 (+2016906us) ts_tablet_manager.cc:1145] Deleting tablet 724d15e05a864852a118badae13a5d3d
0109 20:53:07.328550 (+ 667us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.342448 (+ 13898us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.359088 (+ 16640us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.360249 (+ 1161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2016756}
I20250109 20:53:07.360670 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1342) took 2049 ms. Trace:
I20250109 20:53:07.360759 1157 rpcz_store.cc:276] 0109 20:53:05.311336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.311379 (+ 43us) service_pool.cc:225] Handling call
0109 20:53:07.360570 (+2049191us) ts_tablet_manager.cc:1145] Deleting tablet b0d3e7f598ca4696b733abd94238f85f
0109 20:53:07.360579 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.360658 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2049034}
I20250109 20:53:07.361023 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1343) took 2049 ms. Trace:
I20250109 20:53:07.361112 1157 rpcz_store.cc:276] 0109 20:53:05.311713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.311756 (+ 43us) service_pool.cc:225] Handling call
0109 20:53:07.360943 (+2049187us) ts_tablet_manager.cc:1145] Deleting tablet f954bd7e7f4348d996d24f9157e29fec
0109 20:53:07.360951 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.361012 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2049056}
I20250109 20:53:07.361630 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1345) took 1863 ms. Trace:
I20250109 20:53:07.361729 1157 rpcz_store.cc:276] 0109 20:53:05.498597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.498648 (+ 51us) service_pool.cc:225] Handling call
0109 20:53:07.361335 (+1862687us) ts_tablet_manager.cc:1145] Deleting tablet f9145358b66d489788ec5cb9d4cd68e5
0109 20:53:07.361346 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.361611 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862503}
I20250109 20:53:07.362200 1157 tablet_replica.cc:331] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.362402 1157 raft_consensus.cc:2238] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.362547 1157 raft_consensus.cc:2267] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.362999 1157 ts_tablet_manager.cc:1905] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:07.363058 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6bc95b32aa04e25bbde02bf5e8fe4d9 already in progress: creating tablet
W20250109 20:53:07.363297 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 568e4a83fddd45c8b61849a369074c4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 568e4a83fddd45c8b61849a369074c4d already in progress: creating tablet
I20250109 20:53:07.363297 2533 consensus_queue.cc:1035] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250109 20:53:07.363461 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet aba42482feab4d0f85fb46b4c8b01d19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aba42482feab4d0f85fb46b4c8b01d19 already in progress: creating tablet
W20250109 20:53:07.363608 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 31af7fe649b94a44a1fdf71bdd5fd083 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31af7fe649b94a44a1fdf71bdd5fd083 already in progress: creating tablet
W20250109 20:53:07.363766 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ec1eaa6bc5a84b69840c540319265efb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec1eaa6bc5a84b69840c540319265efb already in progress: creating tablet
W20250109 20:53:07.363940 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5f05e17ca65243a898a7c75d3f453706 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f05e17ca65243a898a7c75d3f453706 already in progress: creating tablet
W20250109 20:53:07.364122 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 908781d94a054386b7d7c06c2c9fa3ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 908781d94a054386b7d7c06c2c9fa3ca already in progress: creating tablet
W20250109 20:53:07.367086 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a13b6eb4563b46aeb6a89faae823f62a (table default.Table_8bc289937bd247a1a05ac1f621324338 [id=28cadf9f4ff64ee59c74d752d52f039f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet a13b6eb4563b46aeb6a89faae823f62a consensus state has no leader
I20250109 20:53:07.372200 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1210 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:53:07.373162 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 724d15e05a864852a118badae13a5d3d (table default.Table_5c9e541186c64e158515aff3e19ccda9 [id=9fcc13c731ac492fbf382b79a3df9ed2]) successfully deleted
W20250109 20:53:07.373389 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b0d3e7f598ca4696b733abd94238f85f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0d3e7f598ca4696b733abd94238f85f already in progress: creating tablet
W20250109 20:53:07.373569 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f954bd7e7f4348d996d24f9157e29fec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f954bd7e7f4348d996d24f9157e29fec already in progress: creating tablet
W20250109 20:53:07.373752 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f9145358b66d489788ec5cb9d4cd68e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9145358b66d489788ec5cb9d4cd68e5 already in progress: creating tablet
I20250109 20:53:07.375341 1160 ts_tablet_manager.cc:1918] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.375444 1160 log.cc:1198] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/daac99da5d10475c9de263bef27baeb7
I20250109 20:53:07.375831 1160 ts_tablet_manager.cc:1939] T daac99da5d10475c9de263bef27baeb7 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.376158 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1260) took 2375 ms. Trace:
I20250109 20:53:07.376266 1160 rpcz_store.cc:276] 0109 20:53:05.000601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.000641 (+ 40us) service_pool.cc:225] Handling call
0109 20:53:07.354758 (+2354117us) ts_tablet_manager.cc:1145] Deleting tablet daac99da5d10475c9de263bef27baeb7
0109 20:53:07.354765 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.372293 (+ 17528us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.375263 (+ 2970us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.376144 (+ 881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2353874}
I20250109 20:53:07.376763 1160 tablet_replica.cc:331] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.376696 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet daac99da5d10475c9de263bef27baeb7 (table default.Table_d70feebb4775482eb8b3b251c50ad92c [id=27a0981b71084436994f1e9689afb145]) successfully deleted
I20250109 20:53:07.377067 1160 raft_consensus.cc:2238] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.377290 1160 raft_consensus.cc:2267] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.377655 1157 ts_tablet_manager.cc:1918] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.378196 1160 ts_tablet_manager.cc:1905] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.378337 1157 log.cc:1198] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/28cd9cd48910497ba3e4b159583cb7a6
I20250109 20:53:07.378630 1157 ts_tablet_manager.cc:1939] T 28cd9cd48910497ba3e4b159583cb7a6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.383610 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1346) took 1874 ms. Trace:
I20250109 20:53:07.383777 1157 rpcz_store.cc:276] 0109 20:53:05.508851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.508903 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:07.362162 (+1853259us) ts_tablet_manager.cc:1145] Deleting tablet 28cd9cd48910497ba3e4b159583cb7a6
0109 20:53:07.362171 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.366072 (+ 3901us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.377574 (+ 11502us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.383585 (+ 6011us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853067}
I20250109 20:53:07.384306 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 28cd9cd48910497ba3e4b159583cb7a6 (table default.Table_49bb1f7179c6420c83e530237052e180 [id=38d4e8c8c6294d3983708d35f536e400]) successfully deleted
I20250109 20:53:07.384651 1157 tablet_replica.cc:331] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.384865 1157 raft_consensus.cc:2238] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.385018 1157 raft_consensus.cc:2267] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.385440 1157 ts_tablet_manager.cc:1905] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.389833 1160 ts_tablet_manager.cc:1918] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.389850 1166 ts_tablet_manager.cc:1918] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.389953 1166 log.cc:1198] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/829bec4f66c24180afd5271057a7d0e8
I20250109 20:53:07.390218 1166 ts_tablet_manager.cc:1939] T 829bec4f66c24180afd5271057a7d0e8 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.390476 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1279) took 2653 ms. Trace:
I20250109 20:53:07.390551 1166 rpcz_store.cc:276] 0109 20:53:04.737480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.741847 (+ 4367us) service_pool.cc:225] Handling call
0109 20:53:07.353438 (+2611591us) ts_tablet_manager.cc:1145] Deleting tablet 829bec4f66c24180afd5271057a7d0e8
0109 20:53:07.353445 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.377372 (+ 23927us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.389776 (+ 12404us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.390462 (+ 686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2611431}
I20250109 20:53:07.390895 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1282) took 2607 ms. Trace:
I20250109 20:53:07.391026 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 829bec4f66c24180afd5271057a7d0e8 (table default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]) successfully deleted
I20250109 20:53:07.392691 1160 log.cc:1198] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/728e9b6101e3447fb3c24b3917aa4872
I20250109 20:53:07.393057 1160 ts_tablet_manager.cc:1939] T 728e9b6101e3447fb3c24b3917aa4872 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.394055 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1261) took 2393 ms. Trace:
I20250109 20:53:07.394160 1160 rpcz_store.cc:276] 0109 20:53:05.000746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.000790 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:07.376699 (+2375909us) ts_tablet_manager.cc:1145] Deleting tablet 728e9b6101e3447fb3c24b3917aa4872
0109 20:53:07.376727 (+ 28us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.387085 (+ 10358us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.389764 (+ 2679us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.394042 (+ 4278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2375767}
I20250109 20:53:07.394541 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 728e9b6101e3447fb3c24b3917aa4872 (table default.Table_6de1d3cd236842a3a239ec4c275d4986 [id=9c304a67389a4bba8058dfabeb20ebcb]) successfully deleted
I20250109 20:53:07.394815 1160 tablet_replica.cc:331] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.395015 1160 raft_consensus.cc:2238] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.396864 910 raft_consensus.cc:3054] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:07.402884 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_159068abc4d842c693e5ae2323d1147d [id=15c9f7d5d3ff4fa4865b55c5892421fb] alter complete (version 3)
I20250109 20:53:07.402954 1160 raft_consensus.cc:2267] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.403400 1160 ts_tablet_manager.cc:1905] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.404578 1157 ts_tablet_manager.cc:1918] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.405236 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 527776a5d742446eb8fe61c3c855021f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.405943 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 3008c672bc544000b24b75ae366d9fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.406296 2145 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" instance_seqno: 1736455961980804) as {username='slave'} at 127.0.104.3:43453; Asking this server to re-register.
I20250109 20:53:07.406867 2475 raft_consensus.cc:491] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.406998 2475 raft_consensus.cc:513] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.407416 2139 ts_manager.cc:194] Registered new tserver with Master: 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:07.407596 2475 leader_election.cc:290] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:07.408471 1101 heartbeater.cc:463] Registering TS with master...
I20250109 20:53:07.409520 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0860554cac64b8882e39fa62462b1aa" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:07.409783 2144 ts_manager.cc:194] Registered new tserver with Master: 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:53:07.410043 989 leader_election.cc:343] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.411290 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5ef3f0f57b0f47d2bd379e2a5710c858"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.411827 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ef3f0f57b0f47d2bd379e2a5710c858 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.408573 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0860554cac64b8882e39fa62462b1aa" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:07.422673 990 leader_election.cc:343] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.422771 990 leader_election.cc:304] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.423194 2475 raft_consensus.cc:2743] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.423388 1157 log.cc:1198] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2ed7abf23ce04a268a460d8e3e97d56b
I20250109 20:53:07.423941 1157 ts_tablet_manager.cc:1939] T 2ed7abf23ce04a268a460d8e3e97d56b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.390967 1166 rpcz_store.cc:276] 0109 20:53:04.783694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.784890 (+ 1196us) service_pool.cc:225] Handling call
0109 20:53:07.390791 (+2605901us) ts_tablet_manager.cc:1145] Deleting tablet 568e4a83fddd45c8b61849a369074c4d
0109 20:53:07.390800 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.390882 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2605686}
W20250109 20:53:07.425601 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 568e4a83fddd45c8b61849a369074c4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 568e4a83fddd45c8b61849a369074c4d already in progress: creating tablet
I20250109 20:53:07.426118 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1286) took 2563 ms. Trace:
I20250109 20:53:07.426216 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1347) took 1917 ms. Trace:
I20250109 20:53:07.426517 1166 rpcz_store.cc:276] 0109 20:53:04.862437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.873646 (+ 11209us) service_pool.cc:225] Handling call
0109 20:53:07.425993 (+2552347us) ts_tablet_manager.cc:1145] Deleting tablet aba42482feab4d0f85fb46b4c8b01d19
0109 20:53:07.426004 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.426092 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2552096}
W20250109 20:53:07.427013 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet aba42482feab4d0f85fb46b4c8b01d19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aba42482feab4d0f85fb46b4c8b01d19 already in progress: creating tablet
I20250109 20:53:07.427663 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1292) took 2498 ms. Trace:
I20250109 20:53:07.428844 1157 rpcz_store.cc:276] 0109 20:53:05.509267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.509317 (+ 50us) service_pool.cc:225] Handling call
0109 20:53:07.384613 (+1875296us) ts_tablet_manager.cc:1145] Deleting tablet 2ed7abf23ce04a268a460d8e3e97d56b
0109 20:53:07.384620 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.391925 (+ 7305us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.404498 (+ 12573us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.426196 (+ 21698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1875152}
I20250109 20:53:07.429522 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2ed7abf23ce04a268a460d8e3e97d56b (table default.Table_422152ed650445aeb8f7af679f9c1d12 [id=b296e8173f88419a8a14ac395611b4bc]) successfully deleted
I20250109 20:53:07.429847 1157 tablet_replica.cc:331] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.430231 1157 raft_consensus.cc:2238] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.430567 1157 raft_consensus.cc:2267] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.431784 1157 ts_tablet_manager.cc:1905] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.432932 1166 rpcz_store.cc:276] 0109 20:53:04.929175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:04.940093 (+ 10918us) service_pool.cc:225] Handling call
0109 20:53:07.427576 (+2487483us) ts_tablet_manager.cc:1145] Deleting tablet 31af7fe649b94a44a1fdf71bdd5fd083
0109 20:53:07.427584 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.427649 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2487211}
W20250109 20:53:07.433310 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 31af7fe649b94a44a1fdf71bdd5fd083 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31af7fe649b94a44a1fdf71bdd5fd083 already in progress: creating tablet
I20250109 20:53:07.433820 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1295) took 2440 ms. Trace:
I20250109 20:53:07.434127 1166 rpcz_store.cc:276] 0109 20:53:04.992882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.003940 (+ 11058us) service_pool.cc:225] Handling call
0109 20:53:07.433730 (+2429790us) ts_tablet_manager.cc:1145] Deleting tablet 908781d94a054386b7d7c06c2c9fa3ca
0109 20:53:07.433738 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.433807 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2429625}
W20250109 20:53:07.434649 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 908781d94a054386b7d7c06c2c9fa3ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 908781d94a054386b7d7c06c2c9fa3ca already in progress: creating tablet
I20250109 20:53:07.435001 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1294) took 2442 ms. Trace:
I20250109 20:53:07.435292 1166 rpcz_store.cc:276] 0109 20:53:04.992770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.005360 (+ 12590us) service_pool.cc:225] Handling call
0109 20:53:07.434918 (+2429558us) ts_tablet_manager.cc:1145] Deleting tablet 5f05e17ca65243a898a7c75d3f453706
0109 20:53:07.434926 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.434988 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2429395}
I20250109 20:53:07.435724 1160 ts_tablet_manager.cc:1918] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250109 20:53:07.435863 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5f05e17ca65243a898a7c75d3f453706 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f05e17ca65243a898a7c75d3f453706 already in progress: creating tablet
I20250109 20:53:07.438375 1160 log.cc:1198] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/8ed134863bf642e79c26e5bf75b248b0
I20250109 20:53:07.438582 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1293) took 2445 ms. Trace:
I20250109 20:53:07.438645 1166 rpcz_store.cc:276] 0109 20:53:04.992653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.005581 (+ 12928us) service_pool.cc:225] Handling call
0109 20:53:07.438489 (+2432908us) ts_tablet_manager.cc:1145] Deleting tablet ec1eaa6bc5a84b69840c540319265efb
0109 20:53:07.438499 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.438568 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2432701}
W20250109 20:53:07.438943 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ec1eaa6bc5a84b69840c540319265efb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec1eaa6bc5a84b69840c540319265efb already in progress: creating tablet
I20250109 20:53:07.440037 1160 ts_tablet_manager.cc:1939] T 8ed134863bf642e79c26e5bf75b248b0 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.440927 1166 tablet_replica.cc:331] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.441280 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1259) took 2440 ms. Trace:
I20250109 20:53:07.441416 1160 rpcz_store.cc:276] 0109 20:53:05.000513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.045976 (+ 45463us) service_pool.cc:225] Handling call
0109 20:53:07.394777 (+2348801us) ts_tablet_manager.cc:1145] Deleting tablet 8ed134863bf642e79c26e5bf75b248b0
0109 20:53:07.394786 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.428337 (+ 33551us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.435628 (+ 7291us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.441261 (+ 5633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347741}
I20250109 20:53:07.441674 1166 raft_consensus.cc:2238] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.441826 1166 raft_consensus.cc:2267] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.442104 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 8ed134863bf642e79c26e5bf75b248b0 (table default.Table_6640cef9559a4ba18d06c0aed445852f [id=46c011650d964e26b40be8a854e1f8c0]) successfully deleted
I20250109 20:53:07.442332 1166 ts_tablet_manager.cc:1905] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.442433 1160 tablet_replica.cc:331] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.442611 1160 raft_consensus.cc:2238] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.443854 1160 raft_consensus.cc:2267] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.444332 1160 ts_tablet_manager.cc:1905] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:07.446802 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19d96d2a15f744f0bee1ad8ab18d16cd consensus state has no leader
I20250109 20:53:07.447786 1157 ts_tablet_manager.cc:1918] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.447942 1157 log.cc:1198] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f03d4e679a1c4793bd83cd979acced88
I20250109 20:53:07.448259 1157 ts_tablet_manager.cc:1939] T f03d4e679a1c4793bd83cd979acced88 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.449216 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1348) took 1928 ms. Trace:
I20250109 20:53:07.449335 1157 rpcz_store.cc:276] 0109 20:53:05.520499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.520610 (+ 111us) service_pool.cc:225] Handling call
0109 20:53:07.429802 (+1909192us) ts_tablet_manager.cc:1145] Deleting tablet f03d4e679a1c4793bd83cd979acced88
0109 20:53:07.429813 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.437733 (+ 7920us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.447700 (+ 9967us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.449198 (+ 1498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1908991}
W20250109 20:53:07.449476 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd8fdc90e32a47e19bdabd30fe2b3525 consensus state has no leader
I20250109 20:53:07.449772 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f03d4e679a1c4793bd83cd979acced88 (table default.Table_7592f0da4beb49cda4e5e9c6392a085b [id=5fbc12ffb04546deb78133a5796c63c6]) successfully deleted
I20250109 20:53:07.450158 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1361) took 1829 ms. Trace:
I20250109 20:53:07.450210 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1210 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:53:07.450263 1157 rpcz_store.cc:276] 0109 20:53:05.621060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.621111 (+ 51us) service_pool.cc:225] Handling call
0109 20:53:07.450038 (+1828927us) ts_tablet_manager.cc:1145] Deleting tablet 530369cc998a4e9980eed6d4b1f57ad4
0109 20:53:07.450046 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.450144 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1828759}
W20250109 20:53:07.450706 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 530369cc998a4e9980eed6d4b1f57ad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 530369cc998a4e9980eed6d4b1f57ad4 already in progress: creating tablet
I20250109 20:53:07.451689 2103 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
I20250109 20:53:07.452178 2615 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250109 20:53:07.452622 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 150313us
I20250109 20:53:07.452698 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1362) took 1831 ms. Trace:
I20250109 20:53:07.452796 1157 rpcz_store.cc:276] 0109 20:53:05.621480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.621524 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:07.452604 (+1831080us) ts_tablet_manager.cc:1145] Deleting tablet 798aa6062f3447419dbaff60ec445b64
0109 20:53:07.452613 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.452684 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830950}
W20250109 20:53:07.453164 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 798aa6062f3447419dbaff60ec445b64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 798aa6062f3447419dbaff60ec445b64 already in progress: creating tablet
I20250109 20:53:07.454548 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_662f7ce5426e4f55b64206655147514a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.454958 1157 tablet_replica.cc:331] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.455086 2556 tablet.cc:1721] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250109 20:53:07.455185 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_662f7ce5426e4f55b64206655147514a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.456360 2597 tablet.cc:1721] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:07.456741 2625 tablet.cc:1721] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:07.455150 1157 raft_consensus.cc:2238] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:07.457160 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4f27bb86556f4fc9a4b88a842fa77edb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.457610 1157 raft_consensus.cc:2267] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:07.457643 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f27bb86556f4fc9a4b88a842fa77edb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.458168 1157 ts_tablet_manager.cc:1905] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.458794 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1a869e33a3be425aa9015a8491edbbd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.459221 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a869e33a3be425aa9015a8491edbbd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.457166 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_43b9440786c14eb0af7ca7e0aadf887b" } new_table_name: "default.Table_44140b18ae054f62b7c95db1da5370f4" modify_external_catalogs: true
I20250109 20:53:07.463068 1166 ts_tablet_manager.cc:1918] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250109 20:53:07.463168 1166 log.cc:1198] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b4b31bd4072941d5aaf8c43dada3ccc7
I20250109 20:53:07.463508 1166 ts_tablet_manager.cc:1939] T b4b31bd4072941d5aaf8c43dada3ccc7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.463770 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1297) took 2393 ms. Trace:
I20250109 20:53:07.463838 1166 rpcz_store.cc:276] 0109 20:53:05.070071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.070835 (+ 764us) service_pool.cc:225] Handling call
0109 20:53:07.440889 (+2370054us) ts_tablet_manager.cc:1145] Deleting tablet b4b31bd4072941d5aaf8c43dada3ccc7
0109 20:53:07.440897 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.460380 (+ 19483us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.462975 (+ 2595us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.463754 (+ 779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2369828}
I20250109 20:53:07.463973 1160 ts_tablet_manager.cc:1918] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250109 20:53:07.464054 1160 log.cc:1198] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/b4b31bd4072941d5aaf8c43dada3ccc7
I20250109 20:53:07.464309 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_8ca6e16543a94e608f9e3a73747ec6d2 [id=2ab3ba8731124704889904608a2f0cd3]) successfully deleted
I20250109 20:53:07.472139 1176 tablet_bootstrap.cc:492] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:07.472260 1165 tablet_bootstrap.cc:654] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.472268 1176 ts_tablet_manager.cc:1397] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.261s user 0.005s sys 0.000s
I20250109 20:53:07.472488 1160 ts_tablet_manager.cc:1939] T b4b31bd4072941d5aaf8c43dada3ccc7 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.472999 1176 raft_consensus.cc:357] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.473213 1176 raft_consensus.cc:383] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.473274 2556 tablet.cc:1721] T 0538f4d3608a454b9dd21a0fa42068a0 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:07.473368 2597 tablet.cc:1721] T 0538f4d3608a454b9dd21a0fa42068a0 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:07.473270 1176 raft_consensus.cc:738] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.474244 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1262) took 2404 ms. Trace:
I20250109 20:53:07.474368 1160 rpcz_store.cc:276] 0109 20:53:05.070286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.073312 (+ 3026us) service_pool.cc:225] Handling call
0109 20:53:07.442349 (+2369037us) ts_tablet_manager.cc:1145] Deleting tablet b4b31bd4072941d5aaf8c43dada3ccc7
0109 20:53:07.442358 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.453568 (+ 11210us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.463884 (+ 10316us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.474226 (+ 10342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368738}
I20250109 20:53:07.475634 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet b4b31bd4072941d5aaf8c43dada3ccc7 (table default.Table_8ca6e16543a94e608f9e3a73747ec6d2 [id=2ab3ba8731124704889904608a2f0cd3]) successfully deleted
I20250109 20:53:07.476130 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1267) took 2390 ms. Trace:
I20250109 20:53:07.476281 1160 rpcz_store.cc:276] 0109 20:53:05.086138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.086376 (+ 238us) service_pool.cc:225] Handling call
0109 20:53:07.475991 (+2389615us) ts_tablet_manager.cc:1145] Deleting tablet 5e843079de154ee5b1b340b2d0d11c7f
0109 20:53:07.476004 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.476110 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2389410}
I20250109 20:53:07.476814 1176 consensus_queue.cc:260] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.477348 1176 ts_tablet_manager.cc:1428] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:53:07.477612 1176 tablet_bootstrap.cc:492] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:53:07.477646 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5e843079de154ee5b1b340b2d0d11c7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e843079de154ee5b1b340b2d0d11c7f already in progress: creating tablet
I20250109 20:53:07.478967 1176 tablet_bootstrap.cc:654] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.479267 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1275) took 2329 ms. Trace:
I20250109 20:53:07.479374 1160 rpcz_store.cc:276] 0109 20:53:05.150244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.151741 (+ 1497us) service_pool.cc:225] Handling call
0109 20:53:07.479148 (+2327407us) ts_tablet_manager.cc:1145] Deleting tablet b54b66ca10114330b3485f67eb076064
0109 20:53:07.479159 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.479250 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2327073}
W20250109 20:53:07.479902 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b54b66ca10114330b3485f67eb076064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54b66ca10114330b3485f67eb076064 already in progress: creating tablet
I20250109 20:53:07.480324 1160 tablet_replica.cc:331] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.480525 1160 raft_consensus.cc:2238] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.480697 1160 raft_consensus.cc:2267] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.481228 1160 ts_tablet_manager.cc:1905] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.481879 2625 tablet.cc:1721] T 0538f4d3608a454b9dd21a0fa42068a0 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:07.483493 1157 ts_tablet_manager.cc:1918] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:07.483609 1157 log.cc:1198] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/d8654c154dab465d8991910698027c3f
I20250109 20:53:07.484233 1157 ts_tablet_manager.cc:1939] T d8654c154dab465d8991910698027c3f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.486057 2552 raft_consensus.cc:491] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.486203 2552 raft_consensus.cc:513] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.486860 2552 leader_election.cc:290] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.486974 2552 raft_consensus.cc:491] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.487099 2552 raft_consensus.cc:513] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.487594 2552 leader_election.cc:290] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.489214 1160 ts_tablet_manager.cc:1918] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.489318 1160 log.cc:1198] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ca816507d0c047668a51f6e7ebce18cb
I20250109 20:53:07.489603 1160 ts_tablet_manager.cc:1939] T ca816507d0c047668a51f6e7ebce18cb P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.489962 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1276) took 2295 ms. Trace:
I20250109 20:53:07.490054 1160 rpcz_store.cc:276] 0109 20:53:05.194743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.194788 (+ 45us) service_pool.cc:225] Handling call
0109 20:53:07.480284 (+2285496us) ts_tablet_manager.cc:1145] Deleting tablet ca816507d0c047668a51f6e7ebce18cb
0109 20:53:07.480291 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.486511 (+ 6220us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.489146 (+ 2635us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.489934 (+ 788us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2285318}
I20250109 20:53:07.490432 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ca816507d0c047668a51f6e7ebce18cb (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]) successfully deleted
I20250109 20:53:07.490697 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1363) took 1868 ms. Trace:
I20250109 20:53:07.491174 1157 rpcz_store.cc:276] 0109 20:53:05.621845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.621891 (+ 46us) service_pool.cc:225] Handling call
0109 20:53:07.454920 (+1833029us) ts_tablet_manager.cc:1145] Deleting tablet d8654c154dab465d8991910698027c3f
0109 20:53:07.454928 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.462017 (+ 7089us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.483417 (+ 21400us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.490675 (+ 7258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1832861}
I20250109 20:53:07.491878 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]) successfully deleted
I20250109 20:53:07.492221 2144 ts_manager.cc:194] Registered new tserver with Master: 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.495050 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1360) took 1874 ms. Trace:
I20250109 20:53:07.495157 1157 rpcz_store.cc:276] 0109 20:53:05.620938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.622920 (+ 1982us) service_pool.cc:225] Handling call
0109 20:53:07.494936 (+1872016us) ts_tablet_manager.cc:1145] Deleting tablet c70ddcc69b2b42fcbe133978dd3ac05b
0109 20:53:07.494946 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.495033 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871878}
I20250109 20:53:07.495344 1165 tablet_bootstrap.cc:492] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.495484 1165 ts_tablet_manager.cc:1397] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.174s user 0.010s sys 0.000s
W20250109 20:53:07.495579 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c70ddcc69b2b42fcbe133978dd3ac05b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c70ddcc69b2b42fcbe133978dd3ac05b already in progress: creating tablet
I20250109 20:53:07.495971 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1372) took 1866 ms. Trace:
I20250109 20:53:07.496083 1157 rpcz_store.cc:276] 0109 20:53:05.629456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.772663 (+143207us) service_pool.cc:225] Handling call
0109 20:53:07.495868 (+1723205us) ts_tablet_manager.cc:1145] Deleting tablet 1e66b7acae984dd6911c3fbf1a06780d
0109 20:53:07.495877 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.495955 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1722992}
I20250109 20:53:07.496362 1165 raft_consensus.cc:357] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.496480 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1e66b7acae984dd6911c3fbf1a06780d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e66b7acae984dd6911c3fbf1a06780d already in progress: creating tablet
I20250109 20:53:07.496562 1165 raft_consensus.cc:383] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.496613 1165 raft_consensus.cc:738] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.496841 1157 tablet_replica.cc:331] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.497061 1157 raft_consensus.cc:2238] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.496975 1165 consensus_queue.cc:260] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.497207 1157 raft_consensus.cc:2267] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.497470 1165 ts_tablet_manager.cc:1428] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:07.497717 1165 tablet_bootstrap.cc:492] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.497699 1157 ts_tablet_manager.cc:1905] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.499455 1165 tablet_bootstrap.cc:654] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.501363 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1277) took 2272 ms. Trace:
I20250109 20:53:07.501525 1160 rpcz_store.cc:276] 0109 20:53:05.228997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.229054 (+ 57us) service_pool.cc:225] Handling call
0109 20:53:07.501251 (+2272197us) ts_tablet_manager.cc:1145] Deleting tablet b0d3e7f598ca4696b733abd94238f85f
0109 20:53:07.501260 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.501348 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2272018}
W20250109 20:53:07.501996 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b0d3e7f598ca4696b733abd94238f85f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0d3e7f598ca4696b733abd94238f85f already in progress: creating tablet
I20250109 20:53:07.502413 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1278) took 2269 ms. Trace:
I20250109 20:53:07.507508 2475 raft_consensus.cc:491] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.507648 2475 raft_consensus.cc:513] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.508276 2475 leader_election.cc:290] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:07.509474 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f407b76780974cf7b3635fa1bf1585ec" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:07.510102 990 leader_election.cc:343] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.509267 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f407b76780974cf7b3635fa1bf1585ec" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:07.511698 2552 raft_consensus.cc:491] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.513886 2552 raft_consensus.cc:513] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.514119 989 leader_election.cc:343] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.514214 989 leader_election.cc:304] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.514439 2552 leader_election.cc:290] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.514516 2475 raft_consensus.cc:2743] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.514695 1160 rpcz_store.cc:276] 0109 20:53:05.232931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.232980 (+ 49us) service_pool.cc:225] Handling call
0109 20:53:07.502296 (+2269316us) ts_tablet_manager.cc:1145] Deleting tablet f954bd7e7f4348d996d24f9157e29fec
0109 20:53:07.502307 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.502395 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2269124}
W20250109 20:53:07.515393 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f954bd7e7f4348d996d24f9157e29fec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f954bd7e7f4348d996d24f9157e29fec already in progress: creating tablet
I20250109 20:53:07.515877 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1279) took 2270 ms. Trace:
I20250109 20:53:07.516248 1160 rpcz_store.cc:276] 0109 20:53:05.245687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.245736 (+ 49us) service_pool.cc:225] Handling call
0109 20:53:07.515761 (+2270025us) ts_tablet_manager.cc:1145] Deleting tablet f9145358b66d489788ec5cb9d4cd68e5
0109 20:53:07.515771 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.515861 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2269838}
W20250109 20:53:07.518296 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f9145358b66d489788ec5cb9d4cd68e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9145358b66d489788ec5cb9d4cd68e5 already in progress: creating tablet
W20250109 20:53:07.511941 854 leader_election.cc:343] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8890a6a93e3d48dc96955f1b51e78603
I20250109 20:53:07.518636 1160 tablet_replica.cc:331] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
W20250109 20:53:07.518771 854 leader_election.cc:343] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 825220629faa4d2481e0db9fa226c306
I20250109 20:53:07.518973 1160 raft_consensus.cc:2238] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.519320 1160 raft_consensus.cc:2267] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.519474 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8414e40b8fb04357b1f98d7ae514207a" } modify_external_catalogs: true
I20250109 20:53:07.519683 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8414e40b8fb04357b1f98d7ae514207a" } modify_external_catalogs: true
I20250109 20:53:07.520238 1160 ts_tablet_manager.cc:1905] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.526042 1165 tablet_bootstrap.cc:492] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.526186 1165 ts_tablet_manager.cc:1397] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20250109 20:53:07.512310 2633 raft_consensus.cc:491] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:53:07.519109 854 leader_election.cc:343] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 8890a6a93e3d48dc96955f1b51e78603
I20250109 20:53:07.526600 854 leader_election.cc:304] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.526934 1165 raft_consensus.cc:357] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.527097 1165 raft_consensus.cc:383] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.527154 1165 raft_consensus.cc:738] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:53:07.527297 854 leader_election.cc:343] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 825220629faa4d2481e0db9fa226c306
I20250109 20:53:07.527308 1165 consensus_queue.cc:260] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.527565 2552 raft_consensus.cc:2743] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.527436 854 leader_election.cc:304] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.527863 1165 ts_tablet_manager.cc:1428] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:07.528018 2552 raft_consensus.cc:2743] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.528327 1165 tablet_bootstrap.cc:492] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.529021 1157 ts_tablet_manager.cc:1918] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.529187 1157 log.cc:1198] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/d57a8bbb3da84088a98c816102e4e9df
W20250109 20:53:07.529605 854 leader_election.cc:343] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: bb0d889cb1114edc8b799b41b23a54a6
W20250109 20:53:07.530014 854 leader_election.cc:343] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: bb0d889cb1114edc8b799b41b23a54a6
I20250109 20:53:07.530134 1165 tablet_bootstrap.cc:654] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.530252 854 leader_election.cc:304] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.530568 2552 raft_consensus.cc:2743] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.530944 2633 raft_consensus.cc:513] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.532387 854 leader_election.cc:343] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 59ebeffe516c4c239ae2ee9040276e53
I20250109 20:53:07.526633 1160 ts_tablet_manager.cc:1918] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.532753 1160 log.cc:1198] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/cd23012af5574f37944f9e26c07d05d2
W20250109 20:53:07.533649 854 leader_election.cc:343] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 59ebeffe516c4c239ae2ee9040276e53
I20250109 20:53:07.533785 854 leader_election.cc:304] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.534037 2629 raft_consensus.cc:2743] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.534207 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1301) took 2440 ms. Trace:
I20250109 20:53:07.534279 2633 leader_election.cc:290] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.534333 1166 rpcz_store.cc:276] 0109 20:53:05.093516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.093896 (+ 380us) service_pool.cc:225] Handling call
0109 20:53:07.534049 (+2440153us) ts_tablet_manager.cc:1145] Deleting tablet 5e843079de154ee5b1b340b2d0d11c7f
0109 20:53:07.534059 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.534181 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2439966}
I20250109 20:53:07.534751 1160 ts_tablet_manager.cc:1939] T cd23012af5574f37944f9e26c07d05d2 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:53:07.534888 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5e843079de154ee5b1b340b2d0d11c7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e843079de154ee5b1b340b2d0d11c7f already in progress: creating tablet
I20250109 20:53:07.535378 1166 tablet_replica.cc:331] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.535691 1166 raft_consensus.cc:2238] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.535938 1166 raft_consensus.cc:2267] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.538163 1165 tablet_bootstrap.cc:492] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.538267 1165 ts_tablet_manager.cc:1397] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250109 20:53:07.535161 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1293) took 2086 ms. Trace:
I20250109 20:53:07.538381 1160 rpcz_store.cc:276] 0109 20:53:05.449207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.449263 (+ 56us) service_pool.cc:225] Handling call
0109 20:53:07.518594 (+2069331us) ts_tablet_manager.cc:1145] Deleting tablet cd23012af5574f37944f9e26c07d05d2
0109 20:53:07.518602 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.523646 (+ 5044us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.526557 (+ 2911us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.535144 (+ 8587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2069153}
I20250109 20:53:07.538781 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet cd23012af5574f37944f9e26c07d05d2 (table default.Table_78b865cbf0f2431c9ee05f94caddda22 [id=94d2e550f9da4202ab7c66cdbb2a5941]) successfully deleted
I20250109 20:53:07.538836 1165 raft_consensus.cc:357] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.538980 1165 raft_consensus.cc:383] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.539017 1165 raft_consensus.cc:738] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.539084 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_333a08dce1d74059a4a06ec1e7eb7513 [id=c6a5f3d877dc4165b4db160a774f6d3f] alter complete (version 1)
I20250109 20:53:07.539176 1165 consensus_queue.cc:260] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.539882 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 4597aa225a374487b13647961b1843d8 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:07.540313 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c79e0eceb7b4c6a8c751387a1a7118a" } modify_external_catalogs: true
I20250109 20:53:07.540467 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6c79e0eceb7b4c6a8c751387a1a7118a" } modify_external_catalogs: true
I20250109 20:53:07.539055 1160 tablet_replica.cc:331] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.540793 1160 raft_consensus.cc:2238] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.540961 1160 raft_consensus.cc:2267] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.541127 1157 ts_tablet_manager.cc:1939] T d57a8bbb3da84088a98c816102e4e9df P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.541812 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1373) took 1912 ms. Trace:
I20250109 20:53:07.542064 1160 ts_tablet_manager.cc:1905] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.542388 1166 ts_tablet_manager.cc:1905] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.545279 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 712acab92e0f44a3a585d7d871498aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e020eeb03e041ed977779a07de5bb80) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.538713 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_409e2ace4d204027a7d56be14d3f9c9c" } modify_external_catalogs: true
I20250109 20:53:07.545552 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_409e2ace4d204027a7d56be14d3f9c9c" } modify_external_catalogs: true
I20250109 20:53:07.542447 1165 ts_tablet_manager.cc:1428] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:07.545919 1157 rpcz_store.cc:276] 0109 20:53:05.629570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.772883 (+143313us) service_pool.cc:225] Handling call
0109 20:53:07.496774 (+1723891us) ts_tablet_manager.cc:1145] Deleting tablet d57a8bbb3da84088a98c816102e4e9df
0109 20:53:07.496784 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.500864 (+ 4080us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.528923 (+ 28059us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.541793 (+ 12870us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1723753}
I20250109 20:53:07.546197 1165 tablet_bootstrap.cc:492] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.546466 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet d57a8bbb3da84088a98c816102e4e9df (table default.Table_634a6ecf6a7b4731ae4ac6e03c5375d1 [id=0bf46974692d47c38befe7eca4f23cc9]) successfully deleted
I20250109 20:53:07.546898 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1380) took 1615 ms. Trace:
I20250109 20:53:07.547034 1157 rpcz_store.cc:276] 0109 20:53:05.930972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.932057 (+ 1085us) service_pool.cc:225] Handling call
0109 20:53:07.546757 (+1614700us) ts_tablet_manager.cc:1145] Deleting tablet 5ae81d22bb904903a75590beca4b7ee2
0109 20:53:07.546767 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.546881 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612516}
I20250109 20:53:07.547729 1165 tablet_bootstrap.cc:654] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:07.548080 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5ae81d22bb904903a75590beca4b7ee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae81d22bb904903a75590beca4b7ee2 already in progress: creating tablet
I20250109 20:53:07.548447 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1382) took 1613 ms. Trace:
I20250109 20:53:07.548622 1157 rpcz_store.cc:276] 0109 20:53:05.934625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.934674 (+ 49us) service_pool.cc:225] Handling call
0109 20:53:07.548348 (+1613674us) ts_tablet_manager.cc:1145] Deleting tablet d5e93f3b4c36485a92cfe28d5156b146
0109 20:53:07.548356 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.548431 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612050}
I20250109 20:53:07.548962 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:07.549154 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632 (table default.Table_e05a32da191c46ffb04a2fa03c7920f5 [id=06fb75d530e343559e65bc14300c15b9]). Expected version 1 got 0
I20250109 20:53:07.550697 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db4e85cd4094cf0b5b699402346b980 with delete_type TABLET_DATA_DELETED (replaced by tablet b54b66ca10114330b3485f67eb076064) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.550971 1160 ts_tablet_manager.cc:1918] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:07.551076 1160 log.cc:1198] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/d8654c154dab465d8991910698027c3f
W20250109 20:53:07.553047 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d5e93f3b4c36485a92cfe28d5156b146 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5e93f3b4c36485a92cfe28d5156b146 already in progress: creating tablet
I20250109 20:53:07.553510 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1379) took 1622 ms. Trace:
I20250109 20:53:07.554947 1157 rpcz_store.cc:276] 0109 20:53:05.930897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.931878 (+ 981us) service_pool.cc:225] Handling call
0109 20:53:07.553382 (+1621504us) ts_tablet_manager.cc:1145] Deleting tablet 3fc68008840d4b8fae5185718ab454fa
0109 20:53:07.553394 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.553492 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1616971}
W20250109 20:53:07.555349 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 3fc68008840d4b8fae5185718ab454fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fc68008840d4b8fae5185718ab454fa already in progress: creating tablet
I20250109 20:53:07.555708 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1381) took 1621 ms. Trace:
I20250109 20:53:07.555785 1157 rpcz_store.cc:276] 0109 20:53:05.934535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.935046 (+ 511us) service_pool.cc:225] Handling call
0109 20:53:07.555614 (+1620568us) ts_tablet_manager.cc:1145] Deleting tablet d30d928c44a84263ad93d9ee36b2d632
0109 20:53:07.555622 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.555693 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1616111}
W20250109 20:53:07.556131 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d30d928c44a84263ad93d9ee36b2d632 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d30d928c44a84263ad93d9ee36b2d632 already in progress: creating tablet
I20250109 20:53:07.556399 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1388) took 1597 ms. Trace:
I20250109 20:53:07.556460 1157 rpcz_store.cc:276] 0109 20:53:05.958946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.964824 (+ 5878us) service_pool.cc:225] Handling call
0109 20:53:07.556323 (+1591499us) ts_tablet_manager.cc:1145] Deleting tablet fb57395c2e684e878baf53d76802dc45
0109 20:53:07.556330 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.556387 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1591226}
W20250109 20:53:07.556795 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet fb57395c2e684e878baf53d76802dc45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb57395c2e684e878baf53d76802dc45 already in progress: creating tablet
I20250109 20:53:07.556885 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1385) took 1619 ms. Trace:
I20250109 20:53:07.556936 1157 rpcz_store.cc:276] 0109 20:53:05.937134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.937402 (+ 268us) service_pool.cc:225] Handling call
0109 20:53:07.556790 (+1619388us) ts_tablet_manager.cc:1145] Deleting tablet 4fe6bd7bc23a4d8095f025b01dd5933c
0109 20:53:07.556797 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.556870 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1579612}
W20250109 20:53:07.557209 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4fe6bd7bc23a4d8095f025b01dd5933c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fe6bd7bc23a4d8095f025b01dd5933c already in progress: creating tablet
I20250109 20:53:07.557477 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1f571e8b256c4b74bd690e2d61870d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.558862 1166 ts_tablet_manager.cc:1918] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.558944 1166 log.cc:1198] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/deedac68322e460b8d62adbe849de3f2
W20250109 20:53:07.559146 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f571e8b256c4b74bd690e2d61870d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.559192 1166 ts_tablet_manager.cc:1939] T deedac68322e460b8d62adbe849de3f2 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.559548 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1394) took 1370 ms. Trace:
I20250109 20:53:07.559684 1157 rpcz_store.cc:276] 0109 20:53:06.188765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.190327 (+ 1562us) service_pool.cc:225] Handling call
0109 20:53:07.559463 (+1369136us) ts_tablet_manager.cc:1145] Deleting tablet 8ff04093442e413abe8433e8784b254f
0109 20:53:07.559470 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.559535 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1368988}
I20250109 20:53:07.559967 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1310) took 2447 ms. Trace:
W20250109 20:53:07.560089 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8ff04093442e413abe8433e8784b254f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ff04093442e413abe8433e8784b254f already in progress: creating tablet
I20250109 20:53:07.560047 1166 rpcz_store.cc:276] 0109 20:53:05.112046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.112103 (+ 57us) service_pool.cc:225] Handling call
0109 20:53:07.535339 (+2423236us) ts_tablet_manager.cc:1145] Deleting tablet deedac68322e460b8d62adbe849de3f2
0109 20:53:07.535347 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.544591 (+ 9244us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.558781 (+ 14190us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.559949 (+ 1168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2423067}
I20250109 20:53:07.560420 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1311) took 2410 ms. Trace:
I20250109 20:53:07.560426 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1395) took 1326 ms. Trace:
I20250109 20:53:07.560498 1166 rpcz_store.cc:276] 0109 20:53:05.150484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.150553 (+ 69us) service_pool.cc:225] Handling call
0109 20:53:07.560307 (+2409754us) ts_tablet_manager.cc:1145] Deleting tablet b54b66ca10114330b3485f67eb076064
0109 20:53:07.560316 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.560404 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2409527}
I20250109 20:53:07.560540 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet deedac68322e460b8d62adbe849de3f2 (table default.Table_17aab5c8aa9a4999b2eebd66dcae9f46 [id=3bdc7c0be0d14815a92667bde1011e67]) successfully deleted
I20250109 20:53:07.560550 1157 rpcz_store.cc:276] 0109 20:53:06.234451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.234492 (+ 41us) service_pool.cc:225] Handling call
0109 20:53:07.560349 (+1325857us) ts_tablet_manager.cc:1145] Deleting tablet 49e0863249364194936804ae83645936
0109 20:53:07.560357 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.560414 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1325671}
I20250109 20:53:07.560777 1166 tablet_replica.cc:331] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:53:07.560940 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b54b66ca10114330b3485f67eb076064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54b66ca10114330b3485f67eb076064 already in progress: creating tablet
I20250109 20:53:07.560956 1166 raft_consensus.cc:2238] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.561151 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1396) took 1325 ms. Trace:
W20250109 20:53:07.561161 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 49e0863249364194936804ae83645936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49e0863249364194936804ae83645936 already in progress: creating tablet
I20250109 20:53:07.561209 1166 raft_consensus.cc:2267] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.561260 1157 rpcz_store.cc:276] 0109 20:53:06.235351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.235399 (+ 48us) service_pool.cc:225] Handling call
0109 20:53:07.561070 (+1325671us) ts_tablet_manager.cc:1145] Deleting tablet 7a4b7e008a474978b91426f63dbd96ee
0109 20:53:07.561078 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.561139 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1325424}
W20250109 20:53:07.561640 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7a4b7e008a474978b91426f63dbd96ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a4b7e008a474978b91426f63dbd96ee already in progress: creating tablet
I20250109 20:53:07.561689 1166 ts_tablet_manager.cc:1905] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.561968 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1397) took 1169 ms. Trace:
I20250109 20:53:07.562067 1157 rpcz_store.cc:276] 0109 20:53:06.392798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.393061 (+ 263us) service_pool.cc:225] Handling call
0109 20:53:07.561879 (+1168818us) ts_tablet_manager.cc:1145] Deleting tablet 190ccaff0ca7486f9dac755f2f625677
0109 20:53:07.561887 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.561953 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1168603}
W20250109 20:53:07.562418 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 190ccaff0ca7486f9dac755f2f625677 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 190ccaff0ca7486f9dac755f2f625677 already in progress: creating tablet
I20250109 20:53:07.562734 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1398) took 1161 ms. Trace:
I20250109 20:53:07.562842 1157 rpcz_store.cc:276] 0109 20:53:06.400871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.400923 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:07.562639 (+1161716us) ts_tablet_manager.cc:1145] Deleting tablet 204ae923b8354fc791b5c4fb7197dc76
0109 20:53:07.562646 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.562706 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161558}
W20250109 20:53:07.563225 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 204ae923b8354fc791b5c4fb7197dc76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 204ae923b8354fc791b5c4fb7197dc76 already in progress: creating tablet
I20250109 20:53:07.563503 1157 tablet_replica.cc:331] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.567159 1157 raft_consensus.cc:2238] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.567203 1166 ts_tablet_manager.cc:1918] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.567286 1166 log.cc:1198] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/6461f153a5994412ba05c3b687963fd1
I20250109 20:53:07.567420 1157 raft_consensus.cc:2267] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.567549 1166 ts_tablet_manager.cc:1939] T 6461f153a5994412ba05c3b687963fd1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.568009 1157 ts_tablet_manager.cc:1905] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.568202 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1313) took 2346 ms. Trace:
I20250109 20:53:07.568279 1166 rpcz_store.cc:276] 0109 20:53:05.221853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.221899 (+ 46us) service_pool.cc:225] Handling call
0109 20:53:07.560735 (+2338836us) ts_tablet_manager.cc:1145] Deleting tablet 6461f153a5994412ba05c3b687963fd1
0109 20:53:07.560742 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.564402 (+ 3660us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.567126 (+ 2724us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.568164 (+ 1038us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2338652}
I20250109 20:53:07.568619 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1312) took 2355 ms. Trace:
I20250109 20:53:07.568689 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 6461f153a5994412ba05c3b687963fd1 (table default.Table_81566c8f4bd940aab3c5fca6ab8ff292 [id=37978bfd578840e4962df1552f7db45a]) successfully deleted
I20250109 20:53:07.568691 1166 rpcz_store.cc:276] 0109 20:53:05.213343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.246544 (+ 33201us) service_pool.cc:225] Handling call
0109 20:53:07.568516 (+2321972us) ts_tablet_manager.cc:1145] Deleting tablet b0d3e7f598ca4696b733abd94238f85f
0109 20:53:07.568524 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.568605 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2321798}
I20250109 20:53:07.554037 2625 tablet.cc:1721] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250109 20:53:07.569165 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b0d3e7f598ca4696b733abd94238f85f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0d3e7f598ca4696b733abd94238f85f already in progress: creating tablet
I20250109 20:53:07.554158 2632 raft_consensus.cc:491] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09833ea46b7141d58c3899dace69e49f)
I20250109 20:53:07.569411 1160 ts_tablet_manager.cc:1939] T d8654c154dab465d8991910698027c3f P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.569461 2632 raft_consensus.cc:513] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.569758 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1295) took 2082 ms. Trace:
I20250109 20:53:07.569844 1160 rpcz_store.cc:276] 0109 20:53:05.487722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.505030 (+ 17308us) service_pool.cc:225] Handling call
0109 20:53:07.539013 (+2033983us) ts_tablet_manager.cc:1145] Deleting tablet d8654c154dab465d8991910698027c3f
0109 20:53:07.539022 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.548123 (+ 9101us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.550841 (+ 2718us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.569739 (+ 18898us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2033779}
I20250109 20:53:07.570271 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]) successfully deleted
I20250109 20:53:07.570331 2632 leader_election.cc:290] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.555684 1165 tablet_bootstrap.cc:492] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.570884 1165 ts_tablet_manager.cc:1397] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20250109 20:53:07.564354 2633 raft_consensus.cc:491] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.571019 2633 raft_consensus.cc:513] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.571599 2633 leader_election.cc:290] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:07.572481 854 leader_election.cc:343] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 72cc0bbbb5584f14ba4d6c62211a12c5
I20250109 20:53:07.572914 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.573675 2413 raft_consensus.cc:491] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.573772 2413 raft_consensus.cc:513] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.574251 2413 leader_election.cc:290] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:07.575101 854 leader_election.cc:343] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 72cc0bbbb5584f14ba4d6c62211a12c5
I20250109 20:53:07.575217 854 leader_election.cc:304] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.575845 2633 raft_consensus.cc:2743] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.576109 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7f67026b164e0faad128d34188afb5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.577544 713 leader_election.cc:343] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: da7f67026b164e0faad128d34188afb5
W20250109 20:53:07.578135 712 leader_election.cc:343] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.578236 712 leader_election.cc:304] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.571533 1165 raft_consensus.cc:357] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.578331 1165 raft_consensus.cc:383] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.578413 1165 raft_consensus.cc:738] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.578541 2533 raft_consensus.cc:2743] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.578604 1165 consensus_queue.cc:260] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.578526 1158 tablet_bootstrap.cc:492] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:07.578995 1158 ts_tablet_manager.cc:1397] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.291s user 0.004s sys 0.012s
I20250109 20:53:07.579078 1165 ts_tablet_manager.cc:1428] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s
I20250109 20:53:07.579563 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1291) took 2154 ms. Trace:
I20250109 20:53:07.579712 1158 raft_consensus.cc:357] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.579720 1160 rpcz_store.cc:276] 0109 20:53:05.425081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.425454 (+ 373us) service_pool.cc:225] Handling call
0109 20:53:07.573753 (+2148299us) ts_tablet_manager.cc:1145] Deleting tablet c70ddcc69b2b42fcbe133978dd3ac05b
0109 20:53:07.579448 (+ 5695us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.579546 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2041729}
I20250109 20:53:07.579912 1158 raft_consensus.cc:383] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.579970 1158 raft_consensus.cc:738] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.580169 1158 consensus_queue.cc:260] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.580348 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c70ddcc69b2b42fcbe133978dd3ac05b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c70ddcc69b2b42fcbe133978dd3ac05b already in progress: creating tablet
I20250109 20:53:07.580677 1158 ts_tablet_manager.cc:1428] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:07.580873 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1292) took 2153 ms. Trace:
I20250109 20:53:07.581003 1160 rpcz_store.cc:276] 0109 20:53:05.426965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.427038 (+ 73us) service_pool.cc:225] Handling call
0109 20:53:07.580649 (+2153611us) ts_tablet_manager.cc:1145] Deleting tablet 530369cc998a4e9980eed6d4b1f57ad4
0109 20:53:07.580659 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.580826 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2047942}
W20250109 20:53:07.581425 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 530369cc998a4e9980eed6d4b1f57ad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 530369cc998a4e9980eed6d4b1f57ad4 already in progress: creating tablet
I20250109 20:53:07.583382 1158 tablet_bootstrap.cc:492] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:07.583613 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1294) took 2113 ms. Trace:
I20250109 20:53:07.583742 1160 rpcz_store.cc:276] 0109 20:53:05.470039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.555782 (+ 85743us) service_pool.cc:225] Handling call
0109 20:53:07.581687 (+2025905us) ts_tablet_manager.cc:1145] Deleting tablet 798aa6062f3447419dbaff60ec445b64
0109 20:53:07.583493 (+ 1806us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.583580 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2025728}
W20250109 20:53:07.584218 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 798aa6062f3447419dbaff60ec445b64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 798aa6062f3447419dbaff60ec445b64 already in progress: creating tablet
I20250109 20:53:07.584565 1165 tablet_bootstrap.cc:492] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.584755 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1304) took 2018 ms. Trace:
I20250109 20:53:07.584916 1158 tablet_bootstrap.cc:654] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.584896 1160 rpcz_store.cc:276] 0109 20:53:05.566707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.566760 (+ 53us) service_pool.cc:225] Handling call
0109 20:53:07.584539 (+2017779us) ts_tablet_manager.cc:1145] Deleting tablet 1e66b7acae984dd6911c3fbf1a06780d
0109 20:53:07.584549 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.584696 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017553}
W20250109 20:53:07.585419 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1e66b7acae984dd6911c3fbf1a06780d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e66b7acae984dd6911c3fbf1a06780d already in progress: creating tablet
I20250109 20:53:07.587301 1165 tablet_bootstrap.cc:654] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.587539 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1311) took 1887 ms. Trace:
I20250109 20:53:07.587692 1160 rpcz_store.cc:276] 0109 20:53:05.699970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.700129 (+ 159us) service_pool.cc:225] Handling call
0109 20:53:07.585703 (+1885574us) ts_tablet_manager.cc:1145] Deleting tablet 3fc68008840d4b8fae5185718ab454fa
0109 20:53:07.587397 (+ 1694us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.587488 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885414}
W20250109 20:53:07.588155 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3fc68008840d4b8fae5185718ab454fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fc68008840d4b8fae5185718ab454fa already in progress: creating tablet
I20250109 20:53:07.588593 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1312) took 1888 ms. Trace:
I20250109 20:53:07.588812 1160 rpcz_store.cc:276] 0109 20:53:05.700082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.700316 (+ 234us) service_pool.cc:225] Handling call
0109 20:53:07.588465 (+1888149us) ts_tablet_manager.cc:1145] Deleting tablet 5ae81d22bb904903a75590beca4b7ee2
0109 20:53:07.588473 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.588551 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1888034}
W20250109 20:53:07.589262 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5ae81d22bb904903a75590beca4b7ee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae81d22bb904903a75590beca4b7ee2 already in progress: creating tablet
I20250109 20:53:07.589639 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1313) took 1881 ms. Trace:
I20250109 20:53:07.589768 1160 rpcz_store.cc:276] 0109 20:53:05.708632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.708706 (+ 74us) service_pool.cc:225] Handling call
0109 20:53:07.589525 (+1880819us) ts_tablet_manager.cc:1145] Deleting tablet d30d928c44a84263ad93d9ee36b2d632
0109 20:53:07.589535 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.589606 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880636}
W20250109 20:53:07.590214 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d30d928c44a84263ad93d9ee36b2d632 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d30d928c44a84263ad93d9ee36b2d632 already in progress: creating tablet
I20250109 20:53:07.600298 1157 ts_tablet_manager.cc:1918] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.600317 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1314) took 1883 ms. Trace:
I20250109 20:53:07.600419 1160 rpcz_store.cc:276] 0109 20:53:05.716568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.716628 (+ 60us) service_pool.cc:225] Handling call
0109 20:53:07.596626 (+1879998us) ts_tablet_manager.cc:1145] Deleting tablet d5e93f3b4c36485a92cfe28d5156b146
0109 20:53:07.600169 (+ 3543us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.600261 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879807}
I20250109 20:53:07.601249 1160 tablet_replica.cc:331] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.601680 1160 raft_consensus.cc:2238] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.602125 1160 raft_consensus.cc:2267] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.569047 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1315) took 2323 ms. Trace:
I20250109 20:53:07.572777 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc32ce3dff0f4d18bb17c6784b0bab41" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:07.602690 1166 rpcz_store.cc:276] 0109 20:53:05.245482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.246740 (+ 1258us) service_pool.cc:225] Handling call
0109 20:53:07.568951 (+2322211us) ts_tablet_manager.cc:1145] Deleting tablet f9145358b66d489788ec5cb9d4cd68e5
0109 20:53:07.568959 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.569033 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2322097}
I20250109 20:53:07.602962 1157 log.cc:1198] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/b74a4c1bdd8f4f6e9592ee5d97c58cc6
I20250109 20:53:07.602963 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:07.603049 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1314) took 2363 ms. Trace:
I20250109 20:53:07.603119 1166 rpcz_store.cc:276] 0109 20:53:05.239187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.263315 (+ 24128us) service_pool.cc:225] Handling call
0109 20:53:07.602930 (+2339615us) ts_tablet_manager.cc:1145] Deleting tablet f954bd7e7f4348d996d24f9157e29fec
0109 20:53:07.602940 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.603030 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2339357}
I20250109 20:53:07.603241 854 leader_election.cc:304] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.603350 1157 ts_tablet_manager.cc:1939] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.603447 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1327) took 2178 ms. Trace:
I20250109 20:53:07.603523 1166 rpcz_store.cc:276] 0109 20:53:05.425350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.425683 (+ 333us) service_pool.cc:225] Handling call
0109 20:53:07.603337 (+2177654us) ts_tablet_manager.cc:1145] Deleting tablet c70ddcc69b2b42fcbe133978dd3ac05b
0109 20:53:07.603345 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.603432 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2177368}
I20250109 20:53:07.603705 2633 raft_consensus.cc:2743] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:53:07.603760 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1328) took 2176 ms. Trace:
I20250109 20:53:07.603814 1166 rpcz_store.cc:276] 0109 20:53:05.427732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.427789 (+ 57us) service_pool.cc:225] Handling call
0109 20:53:07.603677 (+2175888us) ts_tablet_manager.cc:1145] Deleting tablet 530369cc998a4e9980eed6d4b1f57ad4
0109 20:53:07.603684 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.603748 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2175684}
I20250109 20:53:07.604041 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1399) took 1196 ms. Trace:
I20250109 20:53:07.604048 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1329) took 2145 ms. Trace:
I20250109 20:53:07.604120 1166 rpcz_store.cc:276] 0109 20:53:05.458497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.459069 (+ 572us) service_pool.cc:225] Handling call
0109 20:53:07.603967 (+2144898us) ts_tablet_manager.cc:1145] Deleting tablet 798aa6062f3447419dbaff60ec445b64
0109 20:53:07.603973 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.604036 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2144662}
I20250109 20:53:07.604107 1157 rpcz_store.cc:276] 0109 20:53:06.407243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.407299 (+ 56us) service_pool.cc:225] Handling call
0109 20:53:07.563462 (+1156163us) ts_tablet_manager.cc:1145] Deleting tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6
0109 20:53:07.563470 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.593011 (+ 29541us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.600193 (+ 7182us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.604024 (+ 3831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155974}
I20250109 20:53:07.604315 1166 tablet_replica.cc:331] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.604483 1166 raft_consensus.cc:2238] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.604509 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 (table default.Table_7ab416cb67944fa5b70140a8b8ab5e45 [id=5f7ff88269fa48ec80fe384008ba5780]) successfully deleted
I20250109 20:53:07.604640 1166 raft_consensus.cc:2267] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.604858 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1400) took 1174 ms. Trace:
I20250109 20:53:07.604921 1157 rpcz_store.cc:276] 0109 20:53:06.429954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.432943 (+ 2989us) service_pool.cc:225] Handling call
0109 20:53:07.604741 (+1171798us) ts_tablet_manager.cc:1145] Deleting tablet 64d462969643467c97222cac8adc210d
0109 20:53:07.604749 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.604833 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171612}
I20250109 20:53:07.605168 1166 ts_tablet_manager.cc:1905] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.605368 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1401) took 1149 ms. Trace:
I20250109 20:53:07.606392 1160 ts_tablet_manager.cc:1905] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.607985 1157 rpcz_store.cc:276] 0109 20:53:06.455707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.458411 (+ 2704us) service_pool.cc:225] Handling call
0109 20:53:07.605275 (+1146864us) ts_tablet_manager.cc:1145] Deleting tablet 6492caf3355c4685a6f0670ad1a352e9
0109 20:53:07.605282 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.605355 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146627}
I20250109 20:53:07.608546 1157 tablet_replica.cc:331] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:53:07.608618 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 64d462969643467c97222cac8adc210d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d462969643467c97222cac8adc210d already in progress: creating tablet
I20250109 20:53:07.608966 1157 raft_consensus.cc:2238] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:07.608999 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6492caf3355c4685a6f0670ad1a352e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6492caf3355c4685a6f0670ad1a352e9 already in progress: creating tablet
I20250109 20:53:07.609159 1157 raft_consensus.cc:2267] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:07.609217 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f9145358b66d489788ec5cb9d4cd68e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9145358b66d489788ec5cb9d4cd68e5 already in progress: creating tablet
W20250109 20:53:07.609401 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f954bd7e7f4348d996d24f9157e29fec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f954bd7e7f4348d996d24f9157e29fec already in progress: creating tablet
W20250109 20:53:07.609586 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c70ddcc69b2b42fcbe133978dd3ac05b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c70ddcc69b2b42fcbe133978dd3ac05b already in progress: creating tablet
W20250109 20:53:07.609771 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 530369cc998a4e9980eed6d4b1f57ad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 530369cc998a4e9980eed6d4b1f57ad4 already in progress: creating tablet
I20250109 20:53:07.609787 1157 ts_tablet_manager.cc:1905] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:07.609990 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 798aa6062f3447419dbaff60ec445b64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 798aa6062f3447419dbaff60ec445b64 already in progress: creating tablet
W20250109 20:53:07.600960 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d5e93f3b4c36485a92cfe28d5156b146 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5e93f3b4c36485a92cfe28d5156b146 already in progress: creating tablet
I20250109 20:53:07.611295 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:53:07.614935 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_aaeeff82c025462498c8e9d4b583426b" } modify_external_catalogs: true
I20250109 20:53:07.615100 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_aaeeff82c025462498c8e9d4b583426b" } modify_external_catalogs: true
I20250109 20:53:07.615260 1166 ts_tablet_manager.cc:1918] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:07.615338 1166 log.cc:1198] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/d8654c154dab465d8991910698027c3f
I20250109 20:53:07.615667 1166 ts_tablet_manager.cc:1939] T d8654c154dab465d8991910698027c3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.616394 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1330) took 2129 ms. Trace:
I20250109 20:53:07.616473 1166 rpcz_store.cc:276] 0109 20:53:05.487076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.487135 (+ 59us) service_pool.cc:225] Handling call
0109 20:53:07.604280 (+2117145us) ts_tablet_manager.cc:1145] Deleting tablet d8654c154dab465d8991910698027c3f
0109 20:53:07.604287 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.611996 (+ 7709us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.615185 (+ 3189us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.616377 (+ 1192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2116914}
I20250109 20:53:07.616770 1166 tablet_replica.cc:331] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.616950 1166 raft_consensus.cc:2238] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.617055 1166 raft_consensus.cc:2267] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.611815 1165 tablet_bootstrap.cc:492] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.612287 915 raft_consensus.cc:3054] T 4f8419d8cc7e475ea43dd263cae87aca P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:07.617493 1165 ts_tablet_manager.cc:1397] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.004s
I20250109 20:53:07.617592 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_30b1ffb8aacc48c6883ed04b539160e7" } modify_external_catalogs: true
I20250109 20:53:07.617740 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_30b1ffb8aacc48c6883ed04b539160e7" } modify_external_catalogs: true
I20250109 20:53:07.618202 1165 raft_consensus.cc:357] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.618363 1165 raft_consensus.cc:383] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.618420 1165 raft_consensus.cc:738] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.618590 1165 consensus_queue.cc:260] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.619006 1165 ts_tablet_manager.cc:1428] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:07.619077 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2dfa13fd188a4d5481559436ffe57a0a" } new_table_name: "default.Table_1900f241bbde4f10ab2c543802bd6af1" modify_external_catalogs: true
I20250109 20:53:07.619261 1165 tablet_bootstrap.cc:492] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:53:07.620241 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88479a3f57654d84ad622af0782cce47 consensus state has no leader
I20250109 20:53:07.620947 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a1ed66fe3b46456ea784a3471575c052
I20250109 20:53:07.621647 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 40546e1b5e854a79a7f706a43b6c6062
I20250109 20:53:07.621626 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed66fe3b46456ea784a3471575c052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.622571 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 40546e1b5e854a79a7f706a43b6c6062 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.623869 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet d8654c154dab465d8991910698027c3f (table default.Table_ea3048d16571434f9b5a83dadb1160be [id=0b97ce1a3218466ca224e51d62b85577]) successfully deleted
I20250109 20:53:07.624325 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed66fe3b46456ea784a3471575c052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.624429 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 40546e1b5e854a79a7f706a43b6c6062 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.624969 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 40546e1b5e854a79a7f706a43b6c6062 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.624969 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ed66fe3b46456ea784a3471575c052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.626048 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2098d97e88be463c89ad3ea42e26938c
I20250109 20:53:07.627216 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 2098d97e88be463c89ad3ea42e26938c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.627599 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 2098d97e88be463c89ad3ea42e26938c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.629673 1157 ts_tablet_manager.cc:1918] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.632099 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:07.632664 450 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:07.633066 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2098d97e88be463c89ad3ea42e26938c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.633920 1157 log.cc:1198] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/0ec1d30b67674facb8b92a2c1a408073
I20250109 20:53:07.634086 1166 ts_tablet_manager.cc:1905] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.635141 1157 ts_tablet_manager.cc:1939] T 0ec1d30b67674facb8b92a2c1a408073 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.636092 2475 raft_consensus.cc:491] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.636335 1165 tablet_bootstrap.cc:654] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.636268 2475 raft_consensus.cc:513] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.589758 1176 tablet_bootstrap.cc:492] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:07.636900 1176 ts_tablet_manager.cc:1397] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.159s user 0.003s sys 0.000s
I20250109 20:53:07.637048 2475 leader_election.cc:290] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:07.637442 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eaa5bbfaf4c1c9d8cd1b1ead5f632" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.638300 989 leader_election.cc:343] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.638413 1176 raft_consensus.cc:357] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.638715 1176 raft_consensus.cc:383] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.637914 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eaa5bbfaf4c1c9d8cd1b1ead5f632" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.638787 1176 raft_consensus.cc:738] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.639015 1176 consensus_queue.cc:260] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.639276 990 leader_election.cc:343] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.639377 990 leader_election.cc:304] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.639695 1176 ts_tablet_manager.cc:1428] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:53:07.639817 2546 raft_consensus.cc:2743] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.639957 1176 tablet_bootstrap.cc:492] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:07.640226 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1402) took 1176 ms. Trace:
I20250109 20:53:07.640322 1157 rpcz_store.cc:276] 0109 20:53:06.464210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.464283 (+ 73us) service_pool.cc:225] Handling call
0109 20:53:07.608494 (+1144211us) ts_tablet_manager.cc:1145] Deleting tablet 0ec1d30b67674facb8b92a2c1a408073
0109 20:53:07.608506 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.626838 (+ 18332us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.629573 (+ 2735us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.640205 (+ 10632us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144014}
I20250109 20:53:07.640799 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 0ec1d30b67674facb8b92a2c1a408073 (table default.Table_0cbc3b8995704eaf99466fd1d5d0e124 [id=22ffbc424e6d4038a66ddea1fca9842e]) successfully deleted
I20250109 20:53:07.649019 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_038811046f5440acab1ca56d6337e98e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.649649 1157 tablet_replica.cc:331] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
W20250109 20:53:07.649641 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_038811046f5440acab1ca56d6337e98e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.649811 1157 raft_consensus.cc:2238] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.649957 1157 raft_consensus.cc:2267] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.650470 1157 ts_tablet_manager.cc:1905] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.651638 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_71c13b2579c040b69ad859670b25a737" } new_table_name: "default.Table_561754497e6e406394cca03a3b4a0daf" modify_external_catalogs: true
I20250109 20:53:07.653188 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6cba456285b14fd7921032996e6e4c89" } new_table_name: "default.Table_73033d6778e24742a0a23ec4d8d9b39e" modify_external_catalogs: true
I20250109 20:53:07.654589 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0d0e1604420a4a2d9130c7414314649a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.655095 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d0e1604420a4a2d9130c7414314649a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.655717 1157 ts_tablet_manager.cc:1918] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.655812 1157 log.cc:1198] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/bd62a3ae64a143998bb1172b3a97c69f
I20250109 20:53:07.656085 1157 ts_tablet_manager.cc:1939] T bd62a3ae64a143998bb1172b3a97c69f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.657346 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_500eb283bc7f4131b62740a5e293b785" } new_table_name: "default.Table_e6daf126f09444e09bde665034161991" modify_external_catalogs: true
I20250109 20:53:07.660575 1160 ts_tablet_manager.cc:1918] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.665418 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1416) took 1039 ms. Trace:
I20250109 20:53:07.665597 1157 rpcz_store.cc:276] 0109 20:53:06.626024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.626070 (+ 46us) service_pool.cc:225] Handling call
0109 20:53:07.649603 (+1023533us) ts_tablet_manager.cc:1145] Deleting tablet bd62a3ae64a143998bb1172b3a97c69f
0109 20:53:07.649613 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.653082 (+ 3469us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.655643 (+ 2561us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.665399 (+ 9756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1023365}
I20250109 20:53:07.666142 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet bd62a3ae64a143998bb1172b3a97c69f (table default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]) successfully deleted
I20250109 20:53:07.666477 1157 tablet_replica.cc:331] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.666697 1157 raft_consensus.cc:2238] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.666800 1160 log.cc:1198] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/57e56f30e6f3450a8426cd40db89e009
I20250109 20:53:07.666968 1157 raft_consensus.cc:2267] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.667143 1160 ts_tablet_manager.cc:1939] T 57e56f30e6f3450a8426cd40db89e009 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.667481 1157 ts_tablet_manager.cc:1905] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.672122 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1305) took 2007 ms. Trace:
I20250109 20:53:07.672264 1160 rpcz_store.cc:276] 0109 20:53:05.664640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.738898 (+ 74258us) service_pool.cc:225] Handling call
0109 20:53:07.601153 (+1862255us) ts_tablet_manager.cc:1145] Deleting tablet 57e56f30e6f3450a8426cd40db89e009
0109 20:53:07.601163 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.638565 (+ 37402us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.660455 (+ 21890us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.672065 (+ 11610us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1861942}
I20250109 20:53:07.672791 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1317) took 1806 ms. Trace:
I20250109 20:53:07.672789 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 57e56f30e6f3450a8426cd40db89e009 (table default.Table_03765feae81a4760b3e7064beec92515 [id=62d6c3f382274a4bb59feb0a06628572]) successfully deleted
I20250109 20:53:07.672895 1160 rpcz_store.cc:276] 0109 20:53:05.865907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.874586 (+ 8679us) service_pool.cc:225] Handling call
0109 20:53:07.672610 (+1798024us) ts_tablet_manager.cc:1145] Deleting tablet 4fe6bd7bc23a4d8095f025b01dd5933c
0109 20:53:07.672624 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.672752 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1797860}
I20250109 20:53:07.673270 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1320) took 1796 ms. Trace:
W20250109 20:53:07.673316 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4fe6bd7bc23a4d8095f025b01dd5933c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fe6bd7bc23a4d8095f025b01dd5933c already in progress: creating tablet
I20250109 20:53:07.673364 1160 rpcz_store.cc:276] 0109 20:53:05.877254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.903492 (+ 26238us) service_pool.cc:225] Handling call
0109 20:53:07.673143 (+1769651us) ts_tablet_manager.cc:1145] Deleting tablet fb57395c2e684e878baf53d76802dc45
0109 20:53:07.673152 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.673230 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1768101}
W20250109 20:53:07.673707 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet fb57395c2e684e878baf53d76802dc45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb57395c2e684e878baf53d76802dc45 already in progress: creating tablet
I20250109 20:53:07.679939 1157 ts_tablet_manager.cc:1918] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.680089 1157 log.cc:1198] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6c4abc19297e4029b1b7c5396b49b967
I20250109 20:53:07.680372 1157 ts_tablet_manager.cc:1939] T 6c4abc19297e4029b1b7c5396b49b967 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.681227 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1415) took 1055 ms. Trace:
I20250109 20:53:07.682204 1176 tablet_bootstrap.cc:654] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.682317 1157 rpcz_store.cc:276] 0109 20:53:06.625920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.629161 (+ 3241us) service_pool.cc:225] Handling call
0109 20:53:07.666430 (+1037269us) ts_tablet_manager.cc:1145] Deleting tablet 6c4abc19297e4029b1b7c5396b49b967
0109 20:53:07.666440 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.670328 (+ 3888us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.679845 (+ 9517us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.681207 (+ 1362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037078}
I20250109 20:53:07.682570 1166 ts_tablet_manager.cc:1918] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.682662 1166 log.cc:1198] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/72a6cf525cb44c428f7a39a25c4e62dc
I20250109 20:53:07.682955 1166 ts_tablet_manager.cc:1939] T 72a6cf525cb44c428f7a39a25c4e62dc P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.682994 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6c4abc19297e4029b1b7c5396b49b967 (table default.Table_eff4d27a17b3425c92c3d388ee1cc73e [id=6f77cddda4de4f6083cedc70e3026b6d]) successfully deleted
I20250109 20:53:07.683640 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1332) took 2093 ms. Trace:
I20250109 20:53:07.683729 1166 rpcz_store.cc:276] 0109 20:53:05.590603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.591738 (+ 1135us) service_pool.cc:225] Handling call
0109 20:53:07.616688 (+2024950us) ts_tablet_manager.cc:1145] Deleting tablet 72a6cf525cb44c428f7a39a25c4e62dc
0109 20:53:07.616696 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.643732 (+ 27036us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.682479 (+ 38747us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.683620 (+ 1141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2022748}
I20250109 20:53:07.683969 1158 tablet_bootstrap.cc:492] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:07.684121 1158 ts_tablet_manager.cc:1397] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.101s user 0.001s sys 0.002s
I20250109 20:53:07.684110 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1340) took 2089 ms. Trace:
I20250109 20:53:07.684177 1166 rpcz_store.cc:276] 0109 20:53:05.595064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.595517 (+ 453us) service_pool.cc:225] Handling call
0109 20:53:07.683972 (+2088455us) ts_tablet_manager.cc:1145] Deleting tablet 1e66b7acae984dd6911c3fbf1a06780d
0109 20:53:07.683983 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.684086 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2087517}
I20250109 20:53:07.684404 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 72a6cf525cb44c428f7a39a25c4e62dc (table default.Table_5aceb6f20bb841acb677692231b8f670 [id=a662d40e49744d56842c52a82df4df13]) successfully deleted
I20250109 20:53:07.685142 1158 raft_consensus.cc:357] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.685467 1158 raft_consensus.cc:383] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.685598 1158 raft_consensus.cc:738] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.685989 1158 consensus_queue.cc:260] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.686549 1165 tablet_bootstrap.cc:492] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.686704 1165 ts_tablet_manager.cc:1397] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.008s
W20250109 20:53:07.687177 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1b0e82dbd939465ca7044f0aa1f46f06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b0e82dbd939465ca7044f0aa1f46f06 already in progress: creating tablet
I20250109 20:53:07.687520 1157 tablet_replica.cc:331] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.687418 1165 raft_consensus.cc:357] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.687601 1165 raft_consensus.cc:383] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.687670 1165 raft_consensus.cc:738] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.687901 1165 consensus_queue.cc:260] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.688337 1165 ts_tablet_manager.cc:1428] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:07.696514 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_42a10c51a1f649789d65f93ad913becc [id=3fde04a3af42412db45f2c91950fb8e4] alter complete (version 3)
I20250109 20:53:07.697495 1157 raft_consensus.cc:2238] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:07.697811 1157 raft_consensus.cc:2267] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.697825 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1325) took 1509 ms. Trace:
I20250109 20:53:07.697926 1160 rpcz_store.cc:276] 0109 20:53:06.188286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.188354 (+ 68us) service_pool.cc:225] Handling call
0109 20:53:07.673568 (+1485214us) ts_tablet_manager.cc:1145] Deleting tablet 8ff04093442e413abe8433e8784b254f
0109 20:53:07.673576 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.697788 (+ 24212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484969}
I20250109 20:53:07.689173 1158 ts_tablet_manager.cc:1428] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20250109 20:53:07.698357 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8ff04093442e413abe8433e8784b254f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ff04093442e413abe8433e8784b254f already in progress: creating tablet
I20250109 20:53:07.698433 1165 tablet_bootstrap.cc:492] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.699194 455 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 165f6304acc849569a8e90d135a8e2d0 (table default.Table_eac461c564e24066931cc3bb3d22e313 [id=4f170eeebe414cf8a63e96d92ac15320]). Expected version 1 got 0
I20250109 20:53:07.701844 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cc0bbbb5584f14ba4d6c62211a12c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd758fa2fa4cebbdf7faf3856f1941) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.702051 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ed5f7d5524c54a8debe1bc610bd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.702817 1165 tablet_bootstrap.cc:654] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.703294 739 tablet_service.cc:1514] Processing DeleteTablet for tablet da7f67026b164e0faad128d34188afb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.703886 1158 tablet_bootstrap.cc:492] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:07.704931 1166 tablet_replica.cc:331] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.705152 1166 raft_consensus.cc:2238] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.705338 1166 raft_consensus.cc:2267] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:07.705308 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1e66b7acae984dd6911c3fbf1a06780d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e66b7acae984dd6911c3fbf1a06780d already in progress: creating tablet
I20250109 20:53:07.705749 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c88f42c15d94099903178218bfac1e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.706377 1158 tablet_bootstrap.cc:654] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.706530 1157 ts_tablet_manager.cc:1905] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.709137 739 tablet_service.cc:1514] Processing DeleteTablet for tablet dc235614171f4a61b75492ddeb3f0fe6 with delete_type TABLET_DATA_DELETED (replaced by tablet 67be9533983443cbb8fc52031216a973) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.709188 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 404ee892ab79427baaef6d19c57699d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.709558 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1327) took 1506 ms. Trace:
I20250109 20:53:07.709718 1160 rpcz_store.cc:276] 0109 20:53:06.202591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.202642 (+ 51us) service_pool.cc:225] Handling call
0109 20:53:07.709414 (+1506772us) ts_tablet_manager.cc:1145] Deleting tablet 49e0863249364194936804ae83645936
0109 20:53:07.709424 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.709521 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506609}
I20250109 20:53:07.710012 1166 ts_tablet_manager.cc:1905] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:07.710222 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 49e0863249364194936804ae83645936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49e0863249364194936804ae83645936 already in progress: creating tablet
I20250109 20:53:07.710673 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1328) took 1500 ms. Trace:
I20250109 20:53:07.710810 1160 rpcz_store.cc:276] 0109 20:53:06.210674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.210726 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:07.710547 (+1499821us) ts_tablet_manager.cc:1145] Deleting tablet 7a4b7e008a474978b91426f63dbd96ee
0109 20:53:07.710557 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.710638 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499631}
I20250109 20:53:07.713264 1160 tablet_replica.cc:331] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.713703 1160 raft_consensus.cc:2238] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.714015 1160 raft_consensus.cc:2267] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.715247 1160 ts_tablet_manager.cc:1905] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.715480 1166 ts_tablet_manager.cc:1918] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.715588 1166 log.cc:1198] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/0a2efcc2a6ec4b109845d8087491ca9f
I20250109 20:53:07.715749 1157 ts_tablet_manager.cc:1918] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:07.715891 1166 ts_tablet_manager.cc:1939] T 0a2efcc2a6ec4b109845d8087491ca9f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.720772 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 27cb4e6e3a5944bbadb8f1d619d7361c from {username='slave'} at 127.0.0.1:33768
W20250109 20:53:07.713001 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7a4b7e008a474978b91426f63dbd96ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a4b7e008a474978b91426f63dbd96ee already in progress: creating tablet
I20250109 20:53:07.717207 1158 tablet_bootstrap.cc:492] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:07.721208 1158 ts_tablet_manager.cc:1397] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250109 20:53:07.721563 1165 tablet_bootstrap.cc:492] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.721698 1165 ts_tablet_manager.cc:1397] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.001s
I20250109 20:53:07.721962 1158 raft_consensus.cc:357] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.722193 1158 raft_consensus.cc:383] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.722348 1158 raft_consensus.cc:738] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.722381 1165 raft_consensus.cc:357] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.722544 1165 raft_consensus.cc:383] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.722609 1165 raft_consensus.cc:738] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.722748 1158 consensus_queue.cc:260] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.722793 1165 consensus_queue.cc:260] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.723285 1165 ts_tablet_manager.cc:1428] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:07.723781 1165 tablet_bootstrap.cc:492] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.716657 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1341) took 2100 ms. Trace:
I20250109 20:53:07.725783 1166 rpcz_store.cc:276] 0109 20:53:05.616062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.616705 (+ 643us) service_pool.cc:225] Handling call
0109 20:53:07.704873 (+2088168us) ts_tablet_manager.cc:1145] Deleting tablet 0a2efcc2a6ec4b109845d8087491ca9f
0109 20:53:07.704891 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.712869 (+ 7978us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.715409 (+ 2540us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.716639 (+ 1230us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2087888}
I20250109 20:53:07.728140 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet dcba62cbadf04761822bf7161c52b758
I20250109 20:53:07.728470 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 94ed3ab88fd8470bbbc27d801323dc9d
W20250109 20:53:07.728991 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6b23623677b4e088c14af925b4e07d4 consensus state has no leader
I20250109 20:53:07.729341 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9726e60d4f8c433fae0a10b3296d0f8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.730322 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0cf3e606c6e403793fe7ff8e0796efa consensus state has no leader
W20250109 20:53:07.730921 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33a20cac880641d1aaa87b2472fd7978 consensus state has no leader
W20250109 20:53:07.731238 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3974c4196d4043be9ad6e0452ed9cb7b consensus state has no leader
I20250109 20:53:07.732118 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ed3ab88fd8470bbbc27d801323dc9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.733235 1158 ts_tablet_manager.cc:1428] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250109 20:53:07.734045 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1347) took 2030 ms. Trace:
I20250109 20:53:07.734295 1166 rpcz_store.cc:276] 0109 20:53:05.703442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.703624 (+ 182us) service_pool.cc:225] Handling call
0109 20:53:07.726739 (+2023115us) ts_tablet_manager.cc:1145] Deleting tablet 3fc68008840d4b8fae5185718ab454fa
0109 20:53:07.726848 (+ 109us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.734024 (+ 7176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2022937}
I20250109 20:53:07.737633 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1348) took 2034 ms. Trace:
I20250109 20:53:07.737891 1166 rpcz_store.cc:276] 0109 20:53:05.703573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.703624 (+ 51us) service_pool.cc:225] Handling call
0109 20:53:07.737246 (+2033622us) ts_tablet_manager.cc:1145] Deleting tablet 5ae81d22bb904903a75590beca4b7ee2
0109 20:53:07.737256 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.737614 (+ 358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2033422}
I20250109 20:53:07.738406 752 tablet_service.cc:1514] Processing DeleteTablet for tablet c865faed94c3493ea638826bdcd4253d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:07.738443 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3fc68008840d4b8fae5185718ab454fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fc68008840d4b8fae5185718ab454fa already in progress: creating tablet
W20250109 20:53:07.738844 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5ae81d22bb904903a75590beca4b7ee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae81d22bb904903a75590beca4b7ee2 already in progress: creating tablet
I20250109 20:53:07.739042 1158 tablet_bootstrap.cc:492] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:07.738946 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1349) took 1994 ms. Trace:
I20250109 20:53:07.739210 1166 rpcz_store.cc:276] 0109 20:53:05.744098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.744393 (+ 295us) service_pool.cc:225] Handling call
0109 20:53:07.738652 (+1994259us) ts_tablet_manager.cc:1145] Deleting tablet d30d928c44a84263ad93d9ee36b2d632
0109 20:53:07.738661 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.738928 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1993082}
I20250109 20:53:07.739996 1166 tablet_replica.cc:331] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.740599 1158 tablet_bootstrap.cc:654] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.740644 1166 raft_consensus.cc:2238] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.741423 1166 raft_consensus.cc:2267] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.741629 1157 log.cc:1198] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/bc32ce3dff0f4d18bb17c6784b0bab41
I20250109 20:53:07.742035 1157 ts_tablet_manager.cc:1939] T bc32ce3dff0f4d18bb17c6784b0bab41 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.743268 1166 ts_tablet_manager.cc:1905] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:07.743520 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9726e60d4f8c433fae0a10b3296d0f8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.743927 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ed3ab88fd8470bbbc27d801323dc9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.744187 890 tablet_service.cc:1514] Processing DeleteTablet for tablet dcba62cbadf04761822bf7161c52b758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:07.744846 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d30d928c44a84263ad93d9ee36b2d632 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d30d928c44a84263ad93d9ee36b2d632 already in progress: creating tablet
I20250109 20:53:07.745180 1157 tablet_replica.cc:331] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.745330 1157 raft_consensus.cc:2238] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:07.745502 1157 pending_rounds.cc:62] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Trying to abort 1 pending ops.
I20250109 20:53:07.745535 1157 pending_rounds.cc:69] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523722629455872 op_type: NO_OP noop_request { }
I20250109 20:53:07.745640 1157 raft_consensus.cc:2883] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:53:07.745705 1157 raft_consensus.cc:2267] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.746306 1157 ts_tablet_manager.cc:1905] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.746461 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ed3ab88fd8470bbbc27d801323dc9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.746667 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet dcba62cbadf04761822bf7161c52b758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.747367 752 tablet_service.cc:1514] Processing DeleteTablet for tablet dcba62cbadf04761822bf7161c52b758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.747748 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]) successfully deleted
I20250109 20:53:07.749138 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ea477dc845614ea78e71e45f862b57ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.750169 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fd9ac96f54d008e8203e4580e7ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.754354 1158 tablet_bootstrap.cc:492] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:07.754527 1158 ts_tablet_manager.cc:1397] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20250109 20:53:07.755147 1158 raft_consensus.cc:357] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.755359 1158 raft_consensus.cc:383] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.755439 1158 raft_consensus.cc:738] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.755693 1158 consensus_queue.cc:260] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.758322 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_38c4823038d94d6bb4aa5b539be2f3a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.758837 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38c4823038d94d6bb4aa5b539be2f3a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.763895 1158 ts_tablet_manager.cc:1428] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250109 20:53:07.764308 1165 tablet_bootstrap.cc:654] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.766290 1158 tablet_bootstrap.cc:492] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:07.770406 1160 ts_tablet_manager.cc:1918] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.767275 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6569b450bb7a4a78922061d2ede981fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.772397 1158 tablet_bootstrap.cc:654] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.774940 1160 log.cc:1198] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/7da1f3cafffe4f7ca39624b43831689e
I20250109 20:53:07.775425 1160 ts_tablet_manager.cc:1939] T 7da1f3cafffe4f7ca39624b43831689e P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.775897 1157 ts_tablet_manager.cc:1918] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:53:07.776031 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f420b7e623243e093d26c95276c5af9 with delete_type TABLET_DATA_DELETED (replaced by tablet 2420b285905342b3820f9549c8f0c4ad) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.775861 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1326) took 1580 ms. Trace:
I20250109 20:53:07.776314 1160 rpcz_store.cc:276] 0109 20:53:06.195179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.212034 (+ 16855us) service_pool.cc:225] Handling call
0109 20:53:07.713180 (+1501146us) ts_tablet_manager.cc:1145] Deleting tablet 7da1f3cafffe4f7ca39624b43831689e
0109 20:53:07.713191 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.760554 (+ 47363us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.770288 (+ 9734us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.775814 (+ 5526us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1500950}
I20250109 20:53:07.776942 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 7da1f3cafffe4f7ca39624b43831689e (table default.Table_4f4640c9222649fc957636279628a391 [id=cc1214f3eaf54bf0ba8167dbd8a4e634]) successfully deleted
I20250109 20:53:07.777330 1160 tablet_replica.cc:331] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.777588 1160 raft_consensus.cc:2238] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.777824 1160 raft_consensus.cc:2267] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.778420 1160 ts_tablet_manager.cc:1905] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.780790 1157 log.cc:1198] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/048be84b36a1450a818f05fb36cbf3b9
I20250109 20:53:07.781179 1157 ts_tablet_manager.cc:1939] T 048be84b36a1450a818f05fb36cbf3b9 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.782073 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 048be84b36a1450a818f05fb36cbf3b9 (table default.Table_c01a7fe9ae3947a59c88c37c98a59cab [id=adc7cde5fa0445c0a26fafa6c692f8c0]) successfully deleted
W20250109 20:53:07.783291 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4597aa225a374487b13647961b1843d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4597aa225a374487b13647961b1843d8 already in progress: creating tablet
W20250109 20:53:07.784065 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6cb88771d2b148bdae6e427c1b25b9a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cb88771d2b148bdae6e427c1b25b9a9 already in progress: creating tablet
W20250109 20:53:07.784804 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 0f7e3652017946d481090b9940d32e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f7e3652017946d481090b9940d32e97 already in progress: creating tablet
W20250109 20:53:07.785531 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b0dd5ca642034441a0eb5c5a653b7263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0dd5ca642034441a0eb5c5a653b7263 already in progress: creating tablet
W20250109 20:53:07.786231 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet f2d468467478415ba582be6d4a9d989b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d468467478415ba582be6d4a9d989b already in progress: creating tablet
I20250109 20:53:07.786248 1176 tablet_bootstrap.cc:492] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:07.786386 1176 ts_tablet_manager.cc:1397] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.146s user 0.001s sys 0.001s
I20250109 20:53:07.786824 1157 tablet_replica.cc:331] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.787159 1157 raft_consensus.cc:2238] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.787166 1176 raft_consensus.cc:357] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.787359 1176 raft_consensus.cc:383] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.787428 1176 raft_consensus.cc:738] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.787667 1176 consensus_queue.cc:260] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.788090 1157 raft_consensus.cc:2267] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.788163 1176 ts_tablet_manager.cc:1428] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:07.787832 1158 tablet_bootstrap.cc:492] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:07.790683 1158 ts_tablet_manager.cc:1397] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250109 20:53:07.791432 1158 raft_consensus.cc:357] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.791672 1158 raft_consensus.cc:383] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.791764 1158 raft_consensus.cc:738] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.792014 1176 tablet_bootstrap.cc:492] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:07.794129 1176 tablet_bootstrap.cc:654] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.794597 1157 ts_tablet_manager.cc:1905] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.802075 1166 ts_tablet_manager.cc:1918] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.802191 1166 log.cc:1198] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/bdcb6780422541c89343e98e02938d0e
I20250109 20:53:07.802331 2533 raft_consensus.cc:491] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.802439 2533 raft_consensus.cc:513] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.803192 2533 leader_election.cc:290] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.803970 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc235614171f4a61b75492ddeb3f0fe6" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.804476 712 leader_election.cc:343] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:07.805732 713 leader_election.cc:343] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: dc235614171f4a61b75492ddeb3f0fe6
I20250109 20:53:07.805867 713 leader_election.cc:304] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.806231 2533 raft_consensus.cc:2743] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.806306 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ecc48717bbbc4fc1ad0885259f58bc87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.806859 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecc48717bbbc4fc1ad0885259f58bc87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.810739 1160 ts_tablet_manager.cc:1918] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.811017 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 9170b3a6b9bb446b8a474ae4df249409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.811295 2633 raft_consensus.cc:491] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.811453 2633 raft_consensus.cc:513] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.812059 2633 leader_election.cc:290] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:07.813331 854 leader_election.cc:343] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 686ed5f7d5524c54a8debe1bc610bd25
W20250109 20:53:07.813762 854 leader_election.cc:343] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 686ed5f7d5524c54a8debe1bc610bd25
I20250109 20:53:07.813815 1166 ts_tablet_manager.cc:1939] T bdcb6780422541c89343e98e02938d0e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.813853 854 leader_election.cc:304] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.814318 2633 raft_consensus.cc:2743] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.811309 1160 log.cc:1198] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/04e46f4312844b6eb29d6718f4800f98
I20250109 20:53:07.814815 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1351) took 2070 ms. Trace:
I20250109 20:53:07.814841 1160 ts_tablet_manager.cc:1939] T 04e46f4312844b6eb29d6718f4800f98 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.814894 1166 rpcz_store.cc:276] 0109 20:53:05.744316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.758725 (+ 14409us) service_pool.cc:225] Handling call
0109 20:53:07.739847 (+1981122us) ts_tablet_manager.cc:1145] Deleting tablet bdcb6780422541c89343e98e02938d0e
0109 20:53:07.739855 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.774312 (+ 34457us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.801997 (+ 27685us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.814798 (+ 12801us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1980901}
I20250109 20:53:07.815258 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1329) took 1468 ms. Trace:
I20250109 20:53:07.815378 1160 rpcz_store.cc:276] 0109 20:53:06.346894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.346951 (+ 57us) service_pool.cc:225] Handling call
0109 20:53:07.777262 (+1430311us) ts_tablet_manager.cc:1145] Deleting tablet 04e46f4312844b6eb29d6718f4800f98
0109 20:53:07.777272 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.784903 (+ 7631us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.810622 (+ 25719us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.815228 (+ 4606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1396024}
I20250109 20:53:07.815887 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 04e46f4312844b6eb29d6718f4800f98 (table default.Table_c4b9fe08755e4cb1b56978cbda1f2025 [id=4394b742727948f99337b3aa45e81de4]) successfully deleted
I20250109 20:53:07.816337 1160 tablet_replica.cc:331] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.816656 1160 raft_consensus.cc:2238] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.818279 1157 ts_tablet_manager.cc:1918] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.819900 1160 raft_consensus.cc:2267] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.820432 1160 ts_tablet_manager.cc:1905] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.821082 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet bdcb6780422541c89343e98e02938d0e (table default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe [id=fe5e7a3f2800480e860c07f214cfc841]) successfully deleted
I20250109 20:53:07.822752 1157 log.cc:1198] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/006475691733470f8a0468365972dfd2
I20250109 20:53:07.823523 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1350) took 2079 ms. Trace:
I20250109 20:53:07.823597 1166 rpcz_store.cc:276] 0109 20:53:05.744226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.759151 (+ 14925us) service_pool.cc:225] Handling call
0109 20:53:07.815149 (+2055998us) ts_tablet_manager.cc:1145] Deleting tablet d5e93f3b4c36485a92cfe28d5156b146
0109 20:53:07.823392 (+ 8243us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.823507 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2055841}
I20250109 20:53:07.823976 1157 ts_tablet_manager.cc:1939] T 006475691733470f8a0468365972dfd2 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:53:07.824074 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d5e93f3b4c36485a92cfe28d5156b146 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5e93f3b4c36485a92cfe28d5156b146 already in progress: creating tablet
I20250109 20:53:07.827502 1166 tablet_replica.cc:331] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.827762 1166 raft_consensus.cc:2238] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.827885 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 006475691733470f8a0468365972dfd2 (table default.Table_d5db66989ced4d6fb84352f19e087cc7 [id=d65cb2a049bb450391b2f32bb6bbeead]) successfully deleted
I20250109 20:53:07.827929 1166 raft_consensus.cc:2267] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.828061 2475 raft_consensus.cc:491] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.828192 2475 raft_consensus.cc:513] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.828377 1166 ts_tablet_manager.cc:1905] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.828811 2475 leader_election.cc:290] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:07.829377 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527776a5d742446eb8fe61c3c855021f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.829932 989 leader_election.cc:343] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:07.830147 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 086c6b621cae4214bcec4daa1a725272 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 086c6b621cae4214bcec4daa1a725272 already in progress: creating tablet
I20250109 20:53:07.830461 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527776a5d742446eb8fe61c3c855021f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:07.830921 990 leader_election.cc:343] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.831004 990 leader_election.cc:304] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:07.831156 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 94afec96971241cea225c2cf9da1b9c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94afec96971241cea225c2cf9da1b9c5 already in progress: creating tablet
W20250109 20:53:07.831658 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 51b760fcb4fc466ba49d456892d43f3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51b760fcb4fc466ba49d456892d43f3c already in progress: creating tablet
I20250109 20:53:07.831771 2475 raft_consensus.cc:2743] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.832791 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e82bee8a019c4cf4b23febe15b5c5f4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.833340 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e82bee8a019c4cf4b23febe15b5c5f4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.833809 1166 ts_tablet_manager.cc:1918] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.833886 1166 log.cc:1198] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/face19003d0042669e3f22553f9fe19a
I20250109 20:53:07.834151 1166 ts_tablet_manager.cc:1939] T face19003d0042669e3f22553f9fe19a P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.834404 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f02c6d41b4fd4728b4e1501d7bc72dc0" } new_table_name: "default.Table_29a47488e34648d2ab3f4c510af1ad3e" modify_external_catalogs: true
I20250109 20:53:07.835398 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0b103425555a457eb855c29c1f95fb1c" } modify_external_catalogs: true
I20250109 20:53:07.835516 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0b103425555a457eb855c29c1f95fb1c" } modify_external_catalogs: true
I20250109 20:53:07.836560 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d8074d797bd94aaea619f6772e0c644c" } modify_external_catalogs: true
I20250109 20:53:07.836679 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d8074d797bd94aaea619f6772e0c644c" } modify_external_catalogs: true
I20250109 20:53:07.836913 1157 tablet_replica.cc:331] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.837136 1157 raft_consensus.cc:2238] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.837129 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9f9eb96e883d4357b65e2bbb4d2da4bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.837287 1157 raft_consensus.cc:2267] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:07.837498 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f9eb96e883d4357b65e2bbb4d2da4bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.837838 1157 ts_tablet_manager.cc:1905] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.838312 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d2a15480fd94e45985d90ef72fe9070" } modify_external_catalogs: true
I20250109 20:53:07.838418 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1d2a15480fd94e45985d90ef72fe9070" } modify_external_catalogs: true
I20250109 20:53:07.842657 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1352) took 2077 ms. Trace:
I20250109 20:53:07.842738 1166 rpcz_store.cc:276] 0109 20:53:05.764790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.764850 (+ 60us) service_pool.cc:225] Handling call
0109 20:53:07.823832 (+2058982us) ts_tablet_manager.cc:1145] Deleting tablet face19003d0042669e3f22553f9fe19a
0109 20:53:07.827447 (+ 3615us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.831264 (+ 3817us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.833741 (+ 2477us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.842640 (+ 8899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2050674}
I20250109 20:53:07.843068 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1355) took 1977 ms. Trace:
I20250109 20:53:07.843140 1166 rpcz_store.cc:276] 0109 20:53:05.865335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.865421 (+ 86us) service_pool.cc:225] Handling call
0109 20:53:07.842963 (+1977542us) ts_tablet_manager.cc:1145] Deleting tablet 4fe6bd7bc23a4d8095f025b01dd5933c
0109 20:53:07.842971 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.843055 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1977359}
I20250109 20:53:07.843375 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1357) took 1971 ms. Trace:
I20250109 20:53:07.843429 1166 rpcz_store.cc:276] 0109 20:53:05.872238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:05.875417 (+ 3179us) service_pool.cc:225] Handling call
0109 20:53:07.843294 (+1967877us) ts_tablet_manager.cc:1145] Deleting tablet fb57395c2e684e878baf53d76802dc45
0109 20:53:07.843300 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.843364 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967704}
I20250109 20:53:07.843500 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet face19003d0042669e3f22553f9fe19a (table default.Table_94a783531318417989d76ed0dd1d8bdc [id=2b3bd52f32a145fdb6228cf715245ed9]) successfully deleted
I20250109 20:53:07.843654 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1363) took 1655 ms. Trace:
W20250109 20:53:07.843710 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4fe6bd7bc23a4d8095f025b01dd5933c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fe6bd7bc23a4d8095f025b01dd5933c already in progress: creating tablet
I20250109 20:53:07.843701 1166 rpcz_store.cc:276] 0109 20:53:06.188516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.188579 (+ 63us) service_pool.cc:225] Handling call
0109 20:53:07.843573 (+1654994us) ts_tablet_manager.cc:1145] Deleting tablet 8ff04093442e413abe8433e8784b254f
0109 20:53:07.843579 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.843645 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654752}
I20250109 20:53:07.843936 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1364) took 1609 ms. Trace:
I20250109 20:53:07.843994 1166 rpcz_store.cc:276] 0109 20:53:06.234966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.238018 (+ 3052us) service_pool.cc:225] Handling call
0109 20:53:07.843835 (+1605817us) ts_tablet_manager.cc:1145] Deleting tablet 49e0863249364194936804ae83645936
0109 20:53:07.843841 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.843925 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1605165}
W20250109 20:53:07.844127 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet fb57395c2e684e878baf53d76802dc45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb57395c2e684e878baf53d76802dc45 already in progress: creating tablet
I20250109 20:53:07.844238 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1365) took 1608 ms. Trace:
W20250109 20:53:07.844303 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8ff04093442e413abe8433e8784b254f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ff04093442e413abe8433e8784b254f already in progress: creating tablet
I20250109 20:53:07.844287 1166 rpcz_store.cc:276] 0109 20:53:06.235729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.238479 (+ 2750us) service_pool.cc:225] Handling call
0109 20:53:07.844158 (+1605679us) ts_tablet_manager.cc:1145] Deleting tablet 7a4b7e008a474978b91426f63dbd96ee
0109 20:53:07.844164 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.844228 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1605279}
W20250109 20:53:07.844681 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 49e0863249364194936804ae83645936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49e0863249364194936804ae83645936 already in progress: creating tablet
W20250109 20:53:07.844893 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7a4b7e008a474978b91426f63dbd96ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a4b7e008a474978b91426f63dbd96ee already in progress: creating tablet
I20250109 20:53:07.845726 1166 tablet_replica.cc:331] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.845906 1166 raft_consensus.cc:2238] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.846025 1166 raft_consensus.cc:2267] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.846496 1166 ts_tablet_manager.cc:1905] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.846499 1160 ts_tablet_manager.cc:1918] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.846812 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_880e4245498f4e4d9f134497dbe4afd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.847345 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_880e4245498f4e4d9f134497dbe4afd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.848673 1160 log.cc:1198] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/8890a6a93e3d48dc96955f1b51e78603
I20250109 20:53:07.849170 1160 ts_tablet_manager.cc:1939] T 8890a6a93e3d48dc96955f1b51e78603 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.851322 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d914848ad47142f4b64597fd00abdacd
I20250109 20:53:07.851697 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1331) took 1454 ms. Trace:
I20250109 20:53:07.851828 1160 rpcz_store.cc:276] 0109 20:53:06.396848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.396894 (+ 46us) service_pool.cc:225] Handling call
0109 20:53:07.816239 (+1419345us) ts_tablet_manager.cc:1145] Deleting tablet 8890a6a93e3d48dc96955f1b51e78603
0109 20:53:07.816253 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.826744 (+ 10491us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.846388 (+ 19644us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.851667 (+ 5279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419153}
I20250109 20:53:07.852996 739 tablet_service.cc:1514] Processing DeleteTablet for tablet d914848ad47142f4b64597fd00abdacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:07.853665 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet d914848ad47142f4b64597fd00abdacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:07.853945 2475 raft_consensus.cc:491] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.854050 2475 raft_consensus.cc:513] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.854547 2475 leader_election.cc:290] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:07.855551 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c88f42c15d94099903178218bfac1e5" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:07.856073 990 leader_election.cc:343] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.852152 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d914848ad47142f4b64597fd00abdacd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.855091 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c88f42c15d94099903178218bfac1e5" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:07.863247 989 leader_election.cc:343] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.863302 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 8890a6a93e3d48dc96955f1b51e78603 (table default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]) successfully deleted
I20250109 20:53:07.863348 989 leader_election.cc:304] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.863741 2475 raft_consensus.cc:2743] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.863986 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1330) took 1467 ms. Trace:
I20250109 20:53:07.864632 1160 rpcz_store.cc:276] 0109 20:53:06.396710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.396894 (+ 184us) service_pool.cc:225] Handling call
0109 20:53:07.863552 (+1466658us) ts_tablet_manager.cc:1145] Deleting tablet 190ccaff0ca7486f9dac755f2f625677
0109 20:53:07.863562 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.863666 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32,"tablet-delete.queue_time_us":1466472}
W20250109 20:53:07.865633 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 190ccaff0ca7486f9dac755f2f625677 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 190ccaff0ca7486f9dac755f2f625677 already in progress: creating tablet
I20250109 20:53:07.866209 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1332) took 1466 ms. Trace:
I20250109 20:53:07.867377 1166 ts_tablet_manager.cc:1918] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.867465 1166 log.cc:1198] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/1b50c91807734e86840e874886dc0749
I20250109 20:53:07.867748 1166 ts_tablet_manager.cc:1939] T 1b50c91807734e86840e874886dc0749 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.868036 1160 rpcz_store.cc:276] 0109 20:53:06.399969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.400017 (+ 48us) service_pool.cc:225] Handling call
0109 20:53:07.866038 (+1466021us) ts_tablet_manager.cc:1145] Deleting tablet 204ae923b8354fc791b5c4fb7197dc76
0109 20:53:07.866052 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.866155 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465874}
I20250109 20:53:07.868471 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1367) took 1503 ms. Trace:
I20250109 20:53:07.868544 1166 rpcz_store.cc:276] 0109 20:53:06.365310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.365370 (+ 60us) service_pool.cc:225] Handling call
0109 20:53:07.844419 (+1479049us) ts_tablet_manager.cc:1145] Deleting tablet 1b50c91807734e86840e874886dc0749
0109 20:53:07.844425 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.858157 (+ 13732us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.867301 (+ 9144us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.868454 (+ 1153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1478855}
I20250109 20:53:07.869014 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 1b50c91807734e86840e874886dc0749 (table default.Table_189b7740a7c34ecbbec972fa47180efe [id=902f745269374ba6a0c4ecbeeab62bf9]) successfully deleted
W20250109 20:53:07.869122 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 204ae923b8354fc791b5c4fb7197dc76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 204ae923b8354fc791b5c4fb7197dc76 already in progress: creating tablet
I20250109 20:53:07.869678 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1333) took 1437 ms. Trace:
I20250109 20:53:07.871536 1157 ts_tablet_manager.cc:1918] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.871698 1160 rpcz_store.cc:276] 0109 20:53:06.432588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.432649 (+ 61us) service_pool.cc:225] Handling call
0109 20:53:07.869517 (+1436868us) ts_tablet_manager.cc:1145] Deleting tablet 64d462969643467c97222cac8adc210d
0109 20:53:07.869529 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.869627 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436621}
W20250109 20:53:07.872256 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 64d462969643467c97222cac8adc210d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d462969643467c97222cac8adc210d already in progress: creating tablet
I20250109 20:53:07.876966 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1334) took 1416 ms. Trace:
I20250109 20:53:07.877235 1157 log.cc:1198] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/712acab92e0f44a3a585d7d871498aa3
I20250109 20:53:07.877722 1166 tablet_replica.cc:331] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.877626 1165 tablet_bootstrap.cc:492] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.878027 1165 ts_tablet_manager.cc:1397] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.020s
I20250109 20:53:07.877992 1160 rpcz_store.cc:276] 0109 20:53:06.460024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.460076 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:07.876825 (+1416749us) ts_tablet_manager.cc:1145] Deleting tablet 6492caf3355c4685a6f0670ad1a352e9
0109 20:53:07.876843 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.876930 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1416550}
W20250109 20:53:07.878670 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6492caf3355c4685a6f0670ad1a352e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6492caf3355c4685a6f0670ad1a352e9 already in progress: creating tablet
I20250109 20:53:07.877738 1157 ts_tablet_manager.cc:1939] T 712acab92e0f44a3a585d7d871498aa3 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:07.878814 1165 raft_consensus.cc:357] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.879029 1165 raft_consensus.cc:383] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.879105 1165 raft_consensus.cc:738] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.879375 1165 consensus_queue.cc:260] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.879407 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b5f34f72cc4e4e16acdc364a7404e3be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.879874 1165 ts_tablet_manager.cc:1428] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:07.880007 1166 raft_consensus.cc:2238] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.880146 1166 raft_consensus.cc:2267] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.880606 1166 ts_tablet_manager.cc:1905] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:07.882642 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5f34f72cc4e4e16acdc364a7404e3be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.884847 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 712acab92e0f44a3a585d7d871498aa3 (table default.Table_1f4279f07e8b4e0a86928bdb9a67bbf6 [id=8bec14b8b4b74d9c87fe86950f5128e1]) successfully deleted
I20250109 20:53:07.880823 1158 consensus_queue.cc:260] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.887331 1160 tablet_replica.cc:331] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.887514 1160 raft_consensus.cc:2238] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.887703 2633 raft_consensus.cc:491] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.887929 1160 raft_consensus.cc:2267] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.887841 2633 raft_consensus.cc:513] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.888409 2633 leader_election.cc:290] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.888459 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4160bb4ba40347bda48f8fa88f8b6001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.888345 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_696d678105f1412788294191037d4aa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.888947 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_696d678105f1412788294191037d4aa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.889180 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165f6304acc849569a8e90d135a8e2d0" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:07.890566 1160 ts_tablet_manager.cc:1905] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.890754 1158 ts_tablet_manager.cc:1428] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.100s user 0.019s sys 0.000s
I20250109 20:53:07.891044 1165 tablet_bootstrap.cc:492] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.892704 1165 tablet_bootstrap.cc:654] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.898699 1166 ts_tablet_manager.cc:1918] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.898808 1166 log.cc:1198] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/3293f49190684b72a5c41031e621bd36
I20250109 20:53:07.899073 1166 ts_tablet_manager.cc:1939] T 3293f49190684b72a5c41031e621bd36 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.902562 2633 raft_consensus.cc:491] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.902709 2633 raft_consensus.cc:513] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.903345 2633 leader_election.cc:290] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:07.903601 1165 tablet_bootstrap.cc:492] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.903741 1165 ts_tablet_manager.cc:1397] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250109 20:53:07.904426 1165 raft_consensus.cc:357] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.904608 1165 raft_consensus.cc:383] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.904680 1165 raft_consensus.cc:738] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:53:07.904812 854 leader_election.cc:343] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 8f420b7e623243e093d26c95276c5af9
I20250109 20:53:07.904968 1165 consensus_queue.cc:260] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.906358 2556 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:07.908514 1176 tablet_bootstrap.cc:492] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:07.908831 2625 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:07.910852 1165 ts_tablet_manager.cc:1428] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:53:07.911574 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165f6304acc849569a8e90d135a8e2d0" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.914664 854 leader_election.cc:304] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:07.915052 854 leader_election.cc:343] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 8f420b7e623243e093d26c95276c5af9
I20250109 20:53:07.915158 854 leader_election.cc:304] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:07.915349 2632 raft_consensus.cc:2743] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.915496 2632 raft_consensus.cc:2743] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.916365 1165 tablet_bootstrap.cc:492] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.918069 1165 tablet_bootstrap.cc:654] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.921721 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da78917f0b4931ab1705371c1fe5dd with delete_type TABLET_DATA_DELETED (replaced by tablet 3974c4196d4043be9ad6e0452ed9cb7b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:07.925374 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e716270ffcf44039b075023fd0c2d2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:07.928994 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a1ed66fe3b46456ea784a3471575c052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ed66fe3b46456ea784a3471575c052 already in progress: creating tablet
I20250109 20:53:07.933072 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_29a47488e34648d2ab3f4c510af1ad3e [id=1284f08f96184576bde56fa61834e33f] alter complete (version 2)
I20250109 20:53:07.935065 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_603546c0274d4fb49ab48d08448c658d" } new_table_name: "default.Table_d13a0a21f1cd4b9db09a5f9d727cc382" modify_external_catalogs: true
I20250109 20:53:07.935585 1158 tablet_bootstrap.cc:492] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:07.935015 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3e19fcbb0db842afafa6cc97810e53ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:07.936463 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_05225976d35e4c7ea4686a8ed10e7c5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:07.936959 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05225976d35e4c7ea4686a8ed10e7c5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.936988 1158 tablet_bootstrap.cc:654] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:07.937732 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e19fcbb0db842afafa6cc97810e53ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:07.938067 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0446eb38ff3943c098a5979f8bc09fe2" } new_table_name: "default.Table_08b917c79685487ba12b13fec1001b9b" modify_external_catalogs: true
I20250109 20:53:07.939013 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_deeafd87e3e544a4ac810370ae55045e" } new_table_name: "default.Table_462eb45aaf31437ba051a4fbf3d53d43" modify_external_catalogs: true
W20250109 20:53:07.941195 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 40546e1b5e854a79a7f706a43b6c6062 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40546e1b5e854a79a7f706a43b6c6062 already in progress: creating tablet
W20250109 20:53:07.942039 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2098d97e88be463c89ad3ea42e26938c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2098d97e88be463c89ad3ea42e26938c already in progress: creating tablet
I20250109 20:53:07.942286 1176 ts_tablet_manager.cc:1397] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.150s user 0.001s sys 0.003s
I20250109 20:53:07.943209 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c55a378736c64b6ea978572ddc8c6e4e" } new_table_name: "default.Table_be8339044c5443cfb0ea29053fcbb2fd" modify_external_catalogs: true
I20250109 20:53:07.943657 1157 tablet_replica.cc:331] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:07.943874 1157 raft_consensus.cc:2238] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.944322 1176 raft_consensus.cc:357] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.944586 1157 raft_consensus.cc:2267] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.945089 1157 ts_tablet_manager.cc:1905] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.926405 2597 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:07.945530 1176 raft_consensus.cc:383] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.945659 1176 raft_consensus.cc:738] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.946235 1176 consensus_queue.cc:260] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:07.946561 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58f3afa4595a41baa77723f14d6a4944 consensus state has no leader
I20250109 20:53:07.919643 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1368) took 1553 ms. Trace:
I20250109 20:53:07.950181 1166 rpcz_store.cc:276] 0109 20:53:06.365847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.365898 (+ 51us) service_pool.cc:225] Handling call
0109 20:53:07.868884 (+1502986us) ts_tablet_manager.cc:1145] Deleting tablet 3293f49190684b72a5c41031e621bd36
0109 20:53:07.877682 (+ 8798us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.896021 (+ 18339us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.898607 (+ 2586us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.919623 (+ 21016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502828}
I20250109 20:53:07.950546 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1369) took 1558 ms. Trace:
I20250109 20:53:07.950624 1166 rpcz_store.cc:276] 0109 20:53:06.392039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.392356 (+ 317us) service_pool.cc:225] Handling call
0109 20:53:07.950436 (+1558080us) ts_tablet_manager.cc:1145] Deleting tablet 190ccaff0ca7486f9dac755f2f625677
0109 20:53:07.950446 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.950532 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557908}
I20250109 20:53:07.950939 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1370) took 1550 ms. Trace:
I20250109 20:53:07.951015 1166 rpcz_store.cc:276] 0109 20:53:06.400479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.400525 (+ 46us) service_pool.cc:225] Handling call
0109 20:53:07.950799 (+1550274us) ts_tablet_manager.cc:1145] Deleting tablet 204ae923b8354fc791b5c4fb7197dc76
0109 20:53:07.950805 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.950892 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1550133}
I20250109 20:53:07.951301 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1371) took 1520 ms. Trace:
I20250109 20:53:07.951376 1166 rpcz_store.cc:276] 0109 20:53:06.430466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.430542 (+ 76us) service_pool.cc:225] Handling call
0109 20:53:07.951180 (+1520638us) ts_tablet_manager.cc:1145] Deleting tablet 64d462969643467c97222cac8adc210d
0109 20:53:07.951187 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.951258 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1520442}
I20250109 20:53:07.951627 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1372) took 1496 ms. Trace:
I20250109 20:53:07.951696 1166 rpcz_store.cc:276] 0109 20:53:06.455190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.455249 (+ 59us) service_pool.cc:225] Handling call
0109 20:53:07.951532 (+1496283us) ts_tablet_manager.cc:1145] Deleting tablet 6492caf3355c4685a6f0670ad1a352e9
0109 20:53:07.951538 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.951596 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496092}
I20250109 20:53:07.951907 1166 tablet_replica.cc:331] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.952083 1166 raft_consensus.cc:2238] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.952232 1166 raft_consensus.cc:2267] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.953063 1166 ts_tablet_manager.cc:1905] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.953207 1160 ts_tablet_manager.cc:1918] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.953322 1160 log.cc:1198] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2a4f047e8fa34ca79e78915c61a8ba12
I20250109 20:53:07.953610 1160 ts_tablet_manager.cc:1939] T 2a4f047e8fa34ca79e78915c61a8ba12 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.953887 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1335) took 1385 ms. Trace:
I20250109 20:53:07.953963 1160 rpcz_store.cc:276] 0109 20:53:06.568893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.569099 (+ 206us) service_pool.cc:225] Handling call
0109 20:53:07.880578 (+1311479us) ts_tablet_manager.cc:1145] Deleting tablet 2a4f047e8fa34ca79e78915c61a8ba12
0109 20:53:07.887266 (+ 6688us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.934111 (+ 46845us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.953094 (+ 18983us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.953855 (+ 761us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1309901}
I20250109 20:53:07.954335 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2a4f047e8fa34ca79e78915c61a8ba12 (table default.Table_1900f241bbde4f10ab2c543802bd6af1 [id=3ae741f5fe4343188b6f0d10b2a09b69]) successfully deleted
I20250109 20:53:07.955114 1160 tablet_replica.cc:331] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.955293 1160 raft_consensus.cc:2238] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.955426 1160 raft_consensus.cc:2267] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.955902 1160 ts_tablet_manager.cc:1905] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.957367 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 3293f49190684b72a5c41031e621bd36 (table default.Table_2c68d926e1b8418aa0339da1c4cb17cf [id=83feb9b334c14673b257b605634f7bd8]) successfully deleted
W20250109 20:53:07.957589 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 190ccaff0ca7486f9dac755f2f625677 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 190ccaff0ca7486f9dac755f2f625677 already in progress: creating tablet
W20250109 20:53:07.957726 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 204ae923b8354fc791b5c4fb7197dc76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 204ae923b8354fc791b5c4fb7197dc76 already in progress: creating tablet
W20250109 20:53:07.957881 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 64d462969643467c97222cac8adc210d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d462969643467c97222cac8adc210d already in progress: creating tablet
W20250109 20:53:07.958034 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6492caf3355c4685a6f0670ad1a352e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6492caf3355c4685a6f0670ad1a352e9 already in progress: creating tablet
I20250109 20:53:07.961231 1160 ts_tablet_manager.cc:1918] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.961375 1160 log.cc:1198] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/a0d09daa2eed41c3952fa21aa2cdd09e
I20250109 20:53:07.961670 1160 ts_tablet_manager.cc:1939] T a0d09daa2eed41c3952fa21aa2cdd09e P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:07.961954 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1336) took 1392 ms. Trace:
I20250109 20:53:07.962064 1160 rpcz_store.cc:276] 0109 20:53:06.569685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.569728 (+ 43us) service_pool.cc:225] Handling call
0109 20:53:07.954558 (+1384830us) ts_tablet_manager.cc:1145] Deleting tablet a0d09daa2eed41c3952fa21aa2cdd09e
0109 20:53:07.954567 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.958605 (+ 4038us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.961153 (+ 2548us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.961926 (+ 773us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1384666}
I20250109 20:53:07.962222 1166 ts_tablet_manager.cc:1918] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.962339 1166 log.cc:1198] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2797e1df3d12476f89fb5155523c3935
I20250109 20:53:07.962457 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet a0d09daa2eed41c3952fa21aa2cdd09e (table default.Table_14ad6cd05135486e87607f62e9dd62c4 [id=eccfc40a5a774588b25444c9d6f1507d]) successfully deleted
I20250109 20:53:07.962604 1166 ts_tablet_manager.cc:1939] T 2797e1df3d12476f89fb5155523c3935 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:07.962821 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1348) took 1296 ms. Trace:
I20250109 20:53:07.962924 1160 rpcz_store.cc:276] 0109 20:53:06.666272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.666394 (+ 122us) service_pool.cc:225] Handling call
0109 20:53:07.962701 (+1296307us) ts_tablet_manager.cc:1145] Deleting tablet 1b0e82dbd939465ca7044f0aa1f46f06
0109 20:53:07.962710 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.962790 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296114}
I20250109 20:53:07.963289 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1373) took 1388 ms. Trace:
W20250109 20:53:07.963308 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1b0e82dbd939465ca7044f0aa1f46f06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b0e82dbd939465ca7044f0aa1f46f06 already in progress: creating tablet
I20250109 20:53:07.963387 1166 rpcz_store.cc:276] 0109 20:53:06.574870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.574931 (+ 61us) service_pool.cc:225] Handling call
0109 20:53:07.951844 (+1376913us) ts_tablet_manager.cc:1145] Deleting tablet 2797e1df3d12476f89fb5155523c3935
0109 20:53:07.951859 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.956213 (+ 4354us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.962128 (+ 5915us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.963247 (+ 1119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376754}
I20250109 20:53:07.963644 1160 tablet_replica.cc:331] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:07.963678 1166 tablet_replica.cc:331] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:07.963838 1176 ts_tablet_manager.cc:1428] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250109 20:53:07.963842 1160 raft_consensus.cc:2238] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:07.964021 1160 raft_consensus.cc:2267] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:07.964255 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2797e1df3d12476f89fb5155523c3935 (table default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]) successfully deleted
I20250109 20:53:07.964586 1160 ts_tablet_manager.cc:1905] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:07.964951 1176 tablet_bootstrap.cc:492] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:07.966239 1165 tablet_bootstrap.cc:492] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:07.966403 1165 ts_tablet_manager.cc:1397] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.000s
I20250109 20:53:07.966692 2475 raft_consensus.cc:491] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:07.966854 2475 raft_consensus.cc:513] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.967201 1165 raft_consensus.cc:357] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.967368 1165 raft_consensus.cc:383] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.967435 1165 raft_consensus.cc:738] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.967429 2475 leader_election.cc:290] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:07.967661 1165 consensus_queue.cc:260] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.967936 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50039da210df47ab8476e28387c8a802" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:07.968097 1165 ts_tablet_manager.cc:1428] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250109 20:53:07.970748 990 leader_election.cc:343] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.971037 1176 tablet_bootstrap.cc:654] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.971474 1165 tablet_bootstrap.cc:492] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:07.972532 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50039da210df47ab8476e28387c8a802" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:07.973176 1165 tablet_bootstrap.cc:654] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:07.974766 1158 tablet_bootstrap.cc:492] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:07.974905 1158 ts_tablet_manager.cc:1397] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20250109 20:53:07.975446 1158 raft_consensus.cc:357] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.975669 1158 raft_consensus.cc:383] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:07.975756 1158 raft_consensus.cc:738] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:07.976038 1158 consensus_queue.cc:260] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:07.976816 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_735a572f26234eb29120c917e17347c6" } modify_external_catalogs: true
I20250109 20:53:07.976974 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_735a572f26234eb29120c917e17347c6" } modify_external_catalogs: true
I20250109 20:53:07.977399 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_46839e2fe6664d1b9d90e1677fa0d012" } modify_external_catalogs: true
W20250109 20:53:07.977463 989 leader_election.cc:343] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:07.977551 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_46839e2fe6664d1b9d90e1677fa0d012" } modify_external_catalogs: true
I20250109 20:53:07.977602 989 leader_election.cc:304] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:07.977932 2475 raft_consensus.cc:2743] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:07.990456 1160 ts_tablet_manager.cc:1918] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:07.992177 1158 ts_tablet_manager.cc:1428] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250109 20:53:07.999764 1158 tablet_bootstrap.cc:492] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:08.001292 1158 tablet_bootstrap.cc:654] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.001977 1160 log.cc:1198] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/bb0d889cb1114edc8b799b41b23a54a6
I20250109 20:53:08.003440 890 tablet_service.cc:1514] Processing DeleteTablet for tablet e090f72eb1e046319b825ee5fe126332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.005563 1160 ts_tablet_manager.cc:1939] T bb0d889cb1114edc8b799b41b23a54a6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.007743 1166 raft_consensus.cc:2238] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.007926 1166 raft_consensus.cc:2267] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.008450 1166 ts_tablet_manager.cc:1905] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.008529 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1349) took 1294 ms. Trace:
I20250109 20:53:08.009020 1160 rpcz_store.cc:276] 0109 20:53:06.714319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.738269 (+ 23950us) service_pool.cc:225] Handling call
0109 20:53:07.963576 (+1225307us) ts_tablet_manager.cc:1145] Deleting tablet bb0d889cb1114edc8b799b41b23a54a6
0109 20:53:07.963585 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:07.984559 (+ 20974us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:07.990357 (+ 5798us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.008515 (+ 18158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224971}
I20250109 20:53:08.009598 1160 tablet_replica.cc:331] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.009678 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet bb0d889cb1114edc8b799b41b23a54a6 (table default.Table_0bef1709ef9c47738d52c3d41d4f6819 [id=65098b4e94ce440494b2d5682132ba1a]) successfully deleted
I20250109 20:53:08.010035 1160 raft_consensus.cc:2238] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:08.010726 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd6c36d0ccc0463b910c9abbe191b1bd consensus state has no leader
I20250109 20:53:08.010965 1160 raft_consensus.cc:2267] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.012346 1160 ts_tablet_manager.cc:1905] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.014883 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3541107fa6a94c9f9a55103ee6dee809"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.015404 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3541107fa6a94c9f9a55103ee6dee809 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.016363 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 consensus state has no leader
I20250109 20:53:08.016673 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d9bd23a3dfe646688547a78281f08279"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.017084 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9bd23a3dfe646688547a78281f08279 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.017696 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ebaaefdc13e84f2e98f9926278adf637 consensus state has no leader
I20250109 20:53:08.019580 1165 tablet_bootstrap.cc:492] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:08.019690 1165 ts_tablet_manager.cc:1397] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.003s
I20250109 20:53:08.020339 1157 ts_tablet_manager.cc:1918] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.020283 1165 raft_consensus.cc:357] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.020457 1165 raft_consensus.cc:383] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.020510 1165 raft_consensus.cc:738] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.020680 1165 consensus_queue.cc:260] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.021143 1165 ts_tablet_manager.cc:1428] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:53:08.021385 1165 tablet_bootstrap.cc:492] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:08.021517 1157 log.cc:1198] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/da7f67026b164e0faad128d34188afb5
I20250109 20:53:08.021965 1157 ts_tablet_manager.cc:1939] T da7f67026b164e0faad128d34188afb5 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:08.022578 1165 tablet_bootstrap.cc:654] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.024067 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9a6aaac232ee44669f8b733ec7d73b58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.024528 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a6aaac232ee44669f8b733ec7d73b58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.031167 2633 raft_consensus.cc:491] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.031276 2633 raft_consensus.cc:513] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.031736 2633 leader_election.cc:290] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:08.032420 854 leader_election.cc:343] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 246fd9ac96f54d008e8203e4580e7ef3
W20250109 20:53:08.032596 854 leader_election.cc:343] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 246fd9ac96f54d008e8203e4580e7ef3
I20250109 20:53:08.032655 854 leader_election.cc:304] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:08.032991 2633 raft_consensus.cc:2743] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.033179 1157 tablet_replica.cc:331] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:08.033293 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet da7f67026b164e0faad128d34188afb5 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]) successfully deleted
I20250109 20:53:08.033337 1157 raft_consensus.cc:2238] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.033450 1157 raft_consensus.cc:2267] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.036073 1160 ts_tablet_manager.cc:1918] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:08.037277 1176 tablet_bootstrap.cc:492] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.037436 1176 ts_tablet_manager.cc:1397] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.073s user 0.001s sys 0.005s
I20250109 20:53:08.038138 1176 raft_consensus.cc:357] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.038326 1176 raft_consensus.cc:383] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.038406 1176 raft_consensus.cc:738] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.038612 1176 consensus_queue.cc:260] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.043345 1158 tablet_bootstrap.cc:492] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:08.043478 1158 ts_tablet_manager.cc:1397] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20250109 20:53:08.043534 1160 log.cc:1198] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/bc32ce3dff0f4d18bb17c6784b0bab41
I20250109 20:53:08.044039 1160 ts_tablet_manager.cc:1939] T bc32ce3dff0f4d18bb17c6784b0bab41 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.044137 1158 raft_consensus.cc:357] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.044322 1158 raft_consensus.cc:383] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.044377 1158 raft_consensus.cc:738] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.044696 1158 consensus_queue.cc:260] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.044576 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1352) took 1251 ms. Trace:
I20250109 20:53:08.045007 1160 rpcz_store.cc:276] 0109 20:53:06.792940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.800787 (+ 7847us) service_pool.cc:225] Handling call
0109 20:53:08.009555 (+1208768us) ts_tablet_manager.cc:1145] Deleting tablet bc32ce3dff0f4d18bb17c6784b0bab41
0109 20:53:08.009564 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.032700 (+ 23136us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.035992 (+ 3292us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.044561 (+ 8569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1208499}
I20250109 20:53:08.045805 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]) successfully deleted
I20250109 20:53:08.046443 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1351) took 1253 ms. Trace:
I20250109 20:53:08.046576 1160 rpcz_store.cc:276] 0109 20:53:06.792823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.801081 (+ 8258us) service_pool.cc:225] Handling call
0109 20:53:08.046326 (+1245245us) ts_tablet_manager.cc:1145] Deleting tablet 048be84b36a1450a818f05fb36cbf3b9
0109 20:53:08.046333 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.046427 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245099}
W20250109 20:53:08.047209 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 048be84b36a1450a818f05fb36cbf3b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 048be84b36a1450a818f05fb36cbf3b9 already in progress: creating tablet
I20250109 20:53:08.047544 1160 tablet_replica.cc:331] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.047787 1160 raft_consensus.cc:2238] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.047995 1160 raft_consensus.cc:2267] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.048543 1160 ts_tablet_manager.cc:1905] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.050101 1158 ts_tablet_manager.cc:1428] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:53:08.050673 1165 tablet_bootstrap.cc:492] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:08.050782 1165 ts_tablet_manager.cc:1397] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20250109 20:53:08.054042 1158 tablet_bootstrap.cc:492] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:08.056396 1165 raft_consensus.cc:357] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.056579 1165 raft_consensus.cc:383] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.056631 1165 raft_consensus.cc:738] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.056588 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_51b311b735684e74a05c0cd8973a089e" } new_table_name: "default.Table_bbc5532de8774ffc9448c2680924697f" modify_external_catalogs: true
I20250109 20:53:08.058744 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_86f22049bc704f91b332d2205d149706" } modify_external_catalogs: true
I20250109 20:53:08.058890 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_86f22049bc704f91b332d2205d149706" } modify_external_catalogs: true
I20250109 20:53:08.059095 890 tablet_service.cc:1514] Processing DeleteTablet for tablet b68d23fb2df743f4afa2682f6d2e3efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:47 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.056865 1165 consensus_queue.cc:260] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.065709 1165 ts_tablet_manager.cc:1428] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250109 20:53:08.066695 1176 ts_tablet_manager.cc:1428] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250109 20:53:08.067023 1176 tablet_bootstrap.cc:492] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.067768 1165 tablet_bootstrap.cc:492] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:08.068639 1176 tablet_bootstrap.cc:654] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.069090 1165 tablet_bootstrap.cc:654] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.072063 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8f43c379140b4354a8c3fe98c89415f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.072502 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f43c379140b4354a8c3fe98c89415f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.074856 1158 tablet_bootstrap.cc:654] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.074856 1157 ts_tablet_manager.cc:1905] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:08.082171 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6ab69a3204a4c288b369b5319b1ee0d consensus state has no leader
I20250109 20:53:08.083813 1166 ts_tablet_manager.cc:1918] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.083925 1166 log.cc:1198] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/3af9563f6efd47059caac7c86c41d43f
I20250109 20:53:08.083957 2671 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:53:08.084237 1166 ts_tablet_manager.cc:1939] T 3af9563f6efd47059caac7c86c41d43f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:08.084098 2671 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:53:08.084105 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4856edec9677474ea7a18bc9dc730986"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.084576 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4856edec9677474ea7a18bc9dc730986 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.085228 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f85944f66543fe8dd634ad0da0728d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.085423 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1210 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:53:08.085683 2671 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
I20250109 20:53:08.087725 1160 ts_tablet_manager.cc:1918] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.090041 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1210 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:53:08.087831 1160 log.cc:1198] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/e16a1d9a5b374b8dbf725eb32ffadb9e
I20250109 20:53:08.090473 1160 ts_tablet_manager.cc:1939] T e16a1d9a5b374b8dbf725eb32ffadb9e P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.090682 2105 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
I20250109 20:53:08.090742 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1350) took 1298 ms. Trace:
I20250109 20:53:08.090795 1160 rpcz_store.cc:276] 0109 20:53:06.792701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.801642 (+ 8941us) service_pool.cc:225] Handling call
0109 20:53:08.047501 (+1245859us) ts_tablet_manager.cc:1145] Deleting tablet e16a1d9a5b374b8dbf725eb32ffadb9e
0109 20:53:08.047510 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.081704 (+ 34194us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.087638 (+ 5934us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.090728 (+ 3090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245504}
I20250109 20:53:08.091171 2671 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:53:08.091166 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet e16a1d9a5b374b8dbf725eb32ffadb9e (table default.Table_869f0fc9aeaa46e4a95805e5aea7e9b4 [id=fc92038a07574bcf9e5c970ba84eac8c]) successfully deleted
I20250109 20:53:08.091531 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1353) took 1227 ms. Trace:
I20250109 20:53:08.091625 1160 rpcz_store.cc:276] 0109 20:53:06.864166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.864654 (+ 488us) service_pool.cc:225] Handling call
0109 20:53:08.091391 (+1226737us) ts_tablet_manager.cc:1145] Deleting tablet 4597aa225a374487b13647961b1843d8
0109 20:53:08.091401 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.091510 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226540}
W20250109 20:53:08.092068 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4597aa225a374487b13647961b1843d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4597aa225a374487b13647961b1843d8 already in progress: creating tablet
I20250109 20:53:08.092504 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1354) took 1220 ms. Trace:
I20250109 20:53:08.092634 1160 rpcz_store.cc:276] 0109 20:53:06.872316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.872488 (+ 172us) service_pool.cc:225] Handling call
0109 20:53:08.092387 (+1219899us) ts_tablet_manager.cc:1145] Deleting tablet 6cb88771d2b148bdae6e427c1b25b9a9
0109 20:53:08.092399 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.092485 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219735}
W20250109 20:53:08.093115 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6cb88771d2b148bdae6e427c1b25b9a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cb88771d2b148bdae6e427c1b25b9a9 already in progress: creating tablet
I20250109 20:53:08.093525 1160 tablet_replica.cc:331] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.093897 1160 raft_consensus.cc:2238] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.094106 1160 raft_consensus.cc:2267] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.094739 1160 ts_tablet_manager.cc:1905] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.099764 452 master_service.cc:946] ReplaceTablet: received request to replace tablet bd3d0bfb39db40758e82be511d033b4c from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:08.100693 1165 tablet_bootstrap.cc:492] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:08.100852 1165 ts_tablet_manager.cc:1397] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.003s
I20250109 20:53:08.101470 1165 raft_consensus.cc:357] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.101603 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1374) took 1526 ms. Trace:
I20250109 20:53:08.101631 1165 raft_consensus.cc:383] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.101673 1165 raft_consensus.cc:738] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.101688 1166 rpcz_store.cc:276] 0109 20:53:06.574980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.575056 (+ 76us) service_pool.cc:225] Handling call
0109 20:53:07.963624 (+1388568us) ts_tablet_manager.cc:1145] Deleting tablet 3af9563f6efd47059caac7c86c41d43f
0109 20:53:07.963632 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.081143 (+117511us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.083723 (+ 2580us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.101567 (+ 17844us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43852,"tablet-delete.queue_time_us":1388386}
I20250109 20:53:08.102087 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1387) took 1380 ms. Trace:
I20250109 20:53:08.102169 1166 rpcz_store.cc:276] 0109 20:53:06.721575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.721632 (+ 57us) service_pool.cc:225] Handling call
0109 20:53:08.101934 (+1380302us) ts_tablet_manager.cc:1145] Deleting tablet 1b0e82dbd939465ca7044f0aa1f46f06
0109 20:53:08.101946 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.102057 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380106}
I20250109 20:53:08.102442 1166 tablet_replica.cc:331] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:08.102633 1166 raft_consensus.cc:2238] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.102784 1166 raft_consensus.cc:2267] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.103318 1166 ts_tablet_manager.cc:1905] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.101838 1165 consensus_queue.cc:260] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.105474 1165 ts_tablet_manager.cc:1428] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250109 20:53:08.106211 1160 ts_tablet_manager.cc:1918] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.106316 1160 log.cc:1198] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/825220629faa4d2481e0db9fa226c306
I20250109 20:53:08.106633 1160 ts_tablet_manager.cc:1939] T 825220629faa4d2481e0db9fa226c306 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.107483 1176 tablet_bootstrap.cc:492] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.107614 1176 ts_tablet_manager.cc:1397] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250109 20:53:08.108218 1176 raft_consensus.cc:357] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.108383 1176 raft_consensus.cc:383] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.103786 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 3af9563f6efd47059caac7c86c41d43f (table default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3 [id=bb7d5ce383b74f8ea9adc434fd534973]) successfully deleted
W20250109 20:53:08.116609 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1b0e82dbd939465ca7044f0aa1f46f06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b0e82dbd939465ca7044f0aa1f46f06 already in progress: creating tablet
I20250109 20:53:08.119702 915 raft_consensus.cc:3054] T 8e6a9aba1c7b424db21c30b2e92323c9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:08.121042 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1355) took 1111 ms. Trace:
I20250109 20:53:08.121258 1165 tablet_bootstrap.cc:492] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:08.121145 1160 rpcz_store.cc:276] 0109 20:53:07.010052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.010715 (+ 663us) service_pool.cc:225] Handling call
0109 20:53:08.093474 (+1082759us) ts_tablet_manager.cc:1145] Deleting tablet 825220629faa4d2481e0db9fa226c306
0109 20:53:08.093485 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.102519 (+ 9034us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.106122 (+ 3603us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.121020 (+ 14898us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1082578}
I20250109 20:53:08.121479 1160 tablet_replica.cc:331] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.121662 1160 raft_consensus.cc:2238] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.121789 1160 raft_consensus.cc:2267] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.122260 1160 ts_tablet_manager.cc:1905] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.123382 1165 tablet_bootstrap.cc:654] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.125216 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 825220629faa4d2481e0db9fa226c306 (table default.Table_4560626d5c8547678952bb1e7da86ad4 [id=f4c5a8b765814c37882dfdeaf55481e2]) successfully deleted
I20250109 20:53:08.125840 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bd3d0bfb39db40758e82be511d033b4c
I20250109 20:53:08.126729 890 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3d0bfb39db40758e82be511d033b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d1c8c2a0b614f87bc09195c83fac936) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.127298 752 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3d0bfb39db40758e82be511d033b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d1c8c2a0b614f87bc09195c83fac936) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.127319 452 catalog_manager.cc:6596] ReplaceTablet: tablet bd3d0bfb39db40758e82be511d033b4c deleted and replaced by tablet 5d1c8c2a0b614f87bc09195c83fac936
I20250109 20:53:08.127626 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 59ce2173203d4dc4910065f830b335ef from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:08.127771 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3d0bfb39db40758e82be511d033b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d1c8c2a0b614f87bc09195c83fac936) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.128885 2475 raft_consensus.cc:491] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.129025 2475 raft_consensus.cc:513] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.129601 1166 ts_tablet_manager.cc:1918] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.129669 2475 leader_election.cc:290] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:08.130363 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db4e85cd4094cf0b5b699402346b980" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:08.130888 990 leader_election.cc:343] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.132035 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db4e85cd4094cf0b5b699402346b980" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.132457 1160 ts_tablet_manager.cc:1918] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.132542 1160 log.cc:1198] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/14fb1017fe6c4f5abd640ad1f6214232
I20250109 20:53:08.132803 1160 ts_tablet_manager.cc:1939] T 14fb1017fe6c4f5abd640ad1f6214232 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:53:08.133282 989 leader_election.cc:343] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.133399 989 leader_election.cc:304] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.135324 2475 raft_consensus.cc:2743] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.129695 1166 log.cc:1198] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/048be84b36a1450a818f05fb36cbf3b9
I20250109 20:53:08.136279 1166 ts_tablet_manager.cc:1939] T 048be84b36a1450a818f05fb36cbf3b9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:08.136898 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1356) took 1100 ms. Trace:
I20250109 20:53:08.137053 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1390) took 1345 ms. Trace:
I20250109 20:53:08.136996 1160 rpcz_store.cc:276] 0109 20:53:07.036518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.036838 (+ 320us) service_pool.cc:225] Handling call
0109 20:53:08.121438 (+1084600us) ts_tablet_manager.cc:1145] Deleting tablet 14fb1017fe6c4f5abd640ad1f6214232
0109 20:53:08.121446 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.129473 (+ 8027us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.132388 (+ 2915us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.136866 (+ 4478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1084427}
I20250109 20:53:08.137113 1166 rpcz_store.cc:276] 0109 20:53:06.791905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.792072 (+ 167us) service_pool.cc:225] Handling call
0109 20:53:08.102384 (+1310312us) ts_tablet_manager.cc:1145] Deleting tablet 048be84b36a1450a818f05fb36cbf3b9
0109 20:53:08.102391 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.121288 (+ 18897us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.129473 (+ 8185us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.137038 (+ 7565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1310079}
I20250109 20:53:08.137331 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 14fb1017fe6c4f5abd640ad1f6214232 (table default.Table_6c787d10220c41dca63172c9b3bf70ae [id=26365b65a557409abc651578b4458fd4]) successfully deleted
I20250109 20:53:08.137384 1166 tablet_replica.cc:331] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:08.137539 1166 raft_consensus.cc:2238] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.137682 1166 raft_consensus.cc:2267] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.137753 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 048be84b36a1450a818f05fb36cbf3b9 (table default.Table_c01a7fe9ae3947a59c88c37c98a59cab [id=adc7cde5fa0445c0a26fafa6c692f8c0]) successfully deleted
I20250109 20:53:08.140805 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1365) took 1007 ms. Trace:
I20250109 20:53:08.140901 1160 rpcz_store.cc:276] 0109 20:53:07.132888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.132948 (+ 60us) service_pool.cc:225] Handling call
0109 20:53:08.140637 (+1007689us) ts_tablet_manager.cc:1145] Deleting tablet b0dd5ca642034441a0eb5c5a653b7263
0109 20:53:08.140649 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.140789 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007489}
W20250109 20:53:08.141314 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b0dd5ca642034441a0eb5c5a653b7263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0dd5ca642034441a0eb5c5a653b7263 already in progress: creating tablet
I20250109 20:53:08.141410 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:08.141683 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1364) took 1009 ms. Trace:
I20250109 20:53:08.141750 1160 rpcz_store.cc:276] 0109 20:53:07.132118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.135159 (+ 3041us) service_pool.cc:225] Handling call
0109 20:53:08.141575 (+1006416us) ts_tablet_manager.cc:1145] Deleting tablet 0f7e3652017946d481090b9940d32e97
0109 20:53:08.141585 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.141666 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006250}
W20250109 20:53:08.142077 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 0f7e3652017946d481090b9940d32e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f7e3652017946d481090b9940d32e97 already in progress: creating tablet
I20250109 20:53:08.142362 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8801f964bcd24ea5a4c4037abe336954"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:08.142585 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 50039da210df47ab8476e28387c8a802 (table default.Table_422152ed650445aeb8f7af679f9c1d12 [id=b296e8173f88419a8a14ac395611b4bc]). Expected version 3 got 0
W20250109 20:53:08.142880 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8801f964bcd24ea5a4c4037abe336954 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.143164 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3118dded1ed5433a9559f92e920bd70f (DEFAULT_TABLE table=default.Table_f3b6f90f78b545b7973d76289c9a0413 [id=1a949a117d0b471f834f99f5609e759a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.143476 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3118dded1ed5433a9559f92e920bd70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.143402 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5c3d7ff980904543b065367395417f1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.143869 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c3d7ff980904543b065367395417f1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.145603 888 tablet_service.cc:1467] Processing CreateTablet for tablet 18c0394c53314b708aa12343bcd3e0a9 (DEFAULT_TABLE table=default.Table_350b1b2845b4456994e4e3e61ea7fd0b [id=48c4b5b68f3f4aa2b1ae8a139236d485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.145978 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18c0394c53314b708aa12343bcd3e0a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.147451 892 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb6c50c09af432db36609e06a1268a8 (DEFAULT_TABLE table=default.Table_be95c280cf364cbeab60808f1d88975a [id=f2e8c760d27c4602bc0741a82dbd8e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.147744 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb6c50c09af432db36609e06a1268a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.148521 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_11dc31d4cd4b442bb653a6fb17a590ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.148988 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11dc31d4cd4b442bb653a6fb17a590ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.148983 881 tablet_service.cc:1467] Processing CreateTablet for tablet dc74ea9f229e4de3a49b01f2eeec91f0 (DEFAULT_TABLE table=default.Table_db5cb09bc1544977b3d40680b347ac2c [id=bd948f89ac38468caa5dc80f4d24a3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.149257 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc74ea9f229e4de3a49b01f2eeec91f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.149530 1166 ts_tablet_manager.cc:1905] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.153004 884 tablet_service.cc:1467] Processing CreateTablet for tablet 178a93e99e95445d9e240c91cc9199f9 (DEFAULT_TABLE table=default.Table_f5ff906ba8ba4efd971aa6bc36593cf5 [id=3bd1906491054d7dbcaf0c95dd50cd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.153366 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178a93e99e95445d9e240c91cc9199f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.157549 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3118dded1ed5433a9559f92e920bd70f (DEFAULT_TABLE table=default.Table_f3b6f90f78b545b7973d76289c9a0413 [id=1a949a117d0b471f834f99f5609e759a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.157670 739 tablet_service.cc:1467] Processing CreateTablet for tablet 18c0394c53314b708aa12343bcd3e0a9 (DEFAULT_TABLE table=default.Table_350b1b2845b4456994e4e3e61ea7fd0b [id=48c4b5b68f3f4aa2b1ae8a139236d485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.157895 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3118dded1ed5433a9559f92e920bd70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.158172 744 tablet_service.cc:1467] Processing CreateTablet for tablet dc74ea9f229e4de3a49b01f2eeec91f0 (DEFAULT_TABLE table=default.Table_db5cb09bc1544977b3d40680b347ac2c [id=bd948f89ac38468caa5dc80f4d24a3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.158506 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc74ea9f229e4de3a49b01f2eeec91f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.158741 894 tablet_service.cc:1467] Processing CreateTablet for tablet 9615f0d1bb9d46ba8e855fb5819eb5a6 (DEFAULT_TABLE table=default.Table_2bd365ec9d6640318cfafe056f41dbf0 [id=d01ec30decbe41a9a9c802dde0e2402c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.159005 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9615f0d1bb9d46ba8e855fb5819eb5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.160004 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 18c0394c53314b708aa12343bcd3e0a9 (DEFAULT_TABLE table=default.Table_350b1b2845b4456994e4e3e61ea7fd0b [id=48c4b5b68f3f4aa2b1ae8a139236d485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.160398 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18c0394c53314b708aa12343bcd3e0a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.160961 742 tablet_service.cc:1467] Processing CreateTablet for tablet 178a93e99e95445d9e240c91cc9199f9 (DEFAULT_TABLE table=default.Table_f5ff906ba8ba4efd971aa6bc36593cf5 [id=3bd1906491054d7dbcaf0c95dd50cd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.161033 740 tablet_service.cc:1467] Processing CreateTablet for tablet 30e9398f839f4604819e82794af94351 (DEFAULT_TABLE table=default.Table_7a9f34b8d72242458260fc2cc645f85f [id=397fad389c9b45e7bb0a2017841c7a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.161231 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178a93e99e95445d9e240c91cc9199f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.161446 746 tablet_service.cc:1467] Processing CreateTablet for tablet b0a35c4b56f646f6b69d06a45933ff64 (DEFAULT_TABLE table=default.Table_31fe662c8ac540c69f62079a1cf3a9e6 [id=29f2c48d10ea448fbf87835b15589a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.161708 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a35c4b56f646f6b69d06a45933ff64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.162994 886 tablet_service.cc:1467] Processing CreateTablet for tablet a98cb611344f432190685dcca9350219 (DEFAULT_TABLE table=default.Table_e7938ab48958476eadee6c090104823c [id=60fd7f1bce474d3cb4b02bf8b229949f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.162736 882 tablet_service.cc:1467] Processing CreateTablet for tablet b0a35c4b56f646f6b69d06a45933ff64 (DEFAULT_TABLE table=default.Table_31fe662c8ac540c69f62079a1cf3a9e6 [id=29f2c48d10ea448fbf87835b15589a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.163735 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98cb611344f432190685dcca9350219. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.164925 750 tablet_service.cc:1467] Processing CreateTablet for tablet b9e11a8181ae42619a56092017fa9315 (DEFAULT_TABLE table=default.Table_e9289eacd1074dd59e244001042ccbe2 [id=5de4bd9d734647be93e19d743b778ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.164971 1020 tablet_service.cc:1467] Processing CreateTablet for tablet dc74ea9f229e4de3a49b01f2eeec91f0 (DEFAULT_TABLE table=default.Table_db5cb09bc1544977b3d40680b347ac2c [id=bd948f89ac38468caa5dc80f4d24a3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.164971 745 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb6c50c09af432db36609e06a1268a8 (DEFAULT_TABLE table=default.Table_be95c280cf364cbeab60808f1d88975a [id=f2e8c760d27c4602bc0741a82dbd8e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.165179 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e11a8181ae42619a56092017fa9315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.165263 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc74ea9f229e4de3a49b01f2eeec91f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.165416 743 tablet_service.cc:1467] Processing CreateTablet for tablet 9615f0d1bb9d46ba8e855fb5819eb5a6 (DEFAULT_TABLE table=default.Table_2bd365ec9d6640318cfafe056f41dbf0 [id=d01ec30decbe41a9a9c802dde0e2402c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.165731 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9615f0d1bb9d46ba8e855fb5819eb5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.167198 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 178a93e99e95445d9e240c91cc9199f9 (DEFAULT_TABLE table=default.Table_f5ff906ba8ba4efd971aa6bc36593cf5 [id=3bd1906491054d7dbcaf0c95dd50cd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.167506 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178a93e99e95445d9e240c91cc9199f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.168910 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a35c4b56f646f6b69d06a45933ff64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.168973 749 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce5db9c45b24ce7a7731293b3637bb9 (DEFAULT_TABLE table=default.Table_d0aaf004b27e42c8903f69a38b65289f [id=a5cd1a50609847afb1100cdcdfb0508e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.168980 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b0a35c4b56f646f6b69d06a45933ff64 (DEFAULT_TABLE table=default.Table_31fe662c8ac540c69f62079a1cf3a9e6 [id=29f2c48d10ea448fbf87835b15589a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.169284 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce5db9c45b24ce7a7731293b3637bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.169301 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a35c4b56f646f6b69d06a45933ff64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.172235 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 30e9398f839f4604819e82794af94351 (DEFAULT_TABLE table=default.Table_7a9f34b8d72242458260fc2cc645f85f [id=397fad389c9b45e7bb0a2017841c7a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.172540 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e9398f839f4604819e82794af94351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.172971 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb6c50c09af432db36609e06a1268a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.173216 883 tablet_service.cc:1467] Processing CreateTablet for tablet b9e11a8181ae42619a56092017fa9315 (DEFAULT_TABLE table=default.Table_e9289eacd1074dd59e244001042ccbe2 [id=5de4bd9d734647be93e19d743b778ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.173492 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e11a8181ae42619a56092017fa9315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.172917 755 tablet_service.cc:1467] Processing CreateTablet for tablet d9ef0e5255a74d8596dbacdac71c85ca (DEFAULT_TABLE table=default.Table_9234d242e6e44e3494df97820a63cc88 [id=9e02de64790b4fe8bb393842ba201a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.176980 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ef0e5255a74d8596dbacdac71c85ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.177006 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce5db9c45b24ce7a7731293b3637bb9 (DEFAULT_TABLE table=default.Table_d0aaf004b27e42c8903f69a38b65289f [id=a5cd1a50609847afb1100cdcdfb0508e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.177075 757 tablet_service.cc:1467] Processing CreateTablet for tablet eae1da4227e54d17bc98ff68f5c36d4f (DEFAULT_TABLE table=default.Table_e2a32f40c0aa43d8bfb4cc97d297fc76 [id=9f7d9df769fd43e48208451a4ce3718b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.177332 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae1da4227e54d17bc98ff68f5c36d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.177335 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce5db9c45b24ce7a7731293b3637bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.180862 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18c0394c53314b708aa12343bcd3e0a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.180914 887 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce5db9c45b24ce7a7731293b3637bb9 (DEFAULT_TABLE table=default.Table_d0aaf004b27e42c8903f69a38b65289f [id=a5cd1a50609847afb1100cdcdfb0508e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.180946 1024 tablet_service.cc:1467] Processing CreateTablet for tablet a98cb611344f432190685dcca9350219 (DEFAULT_TABLE table=default.Table_e7938ab48958476eadee6c090104823c [id=60fd7f1bce474d3cb4b02bf8b229949f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.181169 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce5db9c45b24ce7a7731293b3637bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.181243 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98cb611344f432190685dcca9350219. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.184975 893 tablet_service.cc:1467] Processing CreateTablet for tablet d9ef0e5255a74d8596dbacdac71c85ca (DEFAULT_TABLE table=default.Table_9234d242e6e44e3494df97820a63cc88 [id=9e02de64790b4fe8bb393842ba201a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.185253 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ef0e5255a74d8596dbacdac71c85ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.185860 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 3118dded1ed5433a9559f92e920bd70f (DEFAULT_TABLE table=default.Table_f3b6f90f78b545b7973d76289c9a0413 [id=1a949a117d0b471f834f99f5609e759a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.186178 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3118dded1ed5433a9559f92e920bd70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.188197 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57642 (request call id 1366) took 1051 ms. Trace:
I20250109 20:53:08.188266 1160 rpcz_store.cc:276] 0109 20:53:07.137018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.138250 (+ 1232us) service_pool.cc:225] Handling call
0109 20:53:08.188089 (+1049839us) ts_tablet_manager.cc:1145] Deleting tablet f2d468467478415ba582be6d4a9d989b
0109 20:53:08.188105 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.188184 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049650}
W20250109 20:53:08.188625 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f2d468467478415ba582be6d4a9d989b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d468467478415ba582be6d4a9d989b already in progress: creating tablet
I20250109 20:53:08.188968 741 tablet_service.cc:1467] Processing CreateTablet for tablet a98cb611344f432190685dcca9350219 (DEFAULT_TABLE table=default.Table_e7938ab48958476eadee6c090104823c [id=60fd7f1bce474d3cb4b02bf8b229949f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.189256 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98cb611344f432190685dcca9350219. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.189291 2633 raft_consensus.cc:491] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.189407 2633 raft_consensus.cc:513] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.190042 2633 leader_election.cc:290] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.192857 895 tablet_service.cc:1467] Processing CreateTablet for tablet eae1da4227e54d17bc98ff68f5c36d4f (DEFAULT_TABLE table=default.Table_e2a32f40c0aa43d8bfb4cc97d297fc76 [id=9f7d9df769fd43e48208451a4ce3718b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.193158 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae1da4227e54d17bc98ff68f5c36d4f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:08.193194 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 94afec96971241cea225c2cf9da1b9c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94afec96971241cea225c2cf9da1b9c5 already in progress: creating tablet
I20250109 20:53:08.193431 1160 tablet_replica.cc:331] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.193596 1160 raft_consensus.cc:2238] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.193722 1160 raft_consensus.cc:2267] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.194173 1160 ts_tablet_manager.cc:1905] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:08.194240 854 leader_election.cc:343] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 4160bb4ba40347bda48f8fa88f8b6001
I20250109 20:53:08.196979 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb6c50c09af432db36609e06a1268a8 (DEFAULT_TABLE table=default.Table_be95c280cf364cbeab60808f1d88975a [id=f2e8c760d27c4602bc0741a82dbd8e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.197283 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb6c50c09af432db36609e06a1268a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.201071 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 9615f0d1bb9d46ba8e855fb5819eb5a6 (DEFAULT_TABLE table=default.Table_2bd365ec9d6640318cfafe056f41dbf0 [id=d01ec30decbe41a9a9c802dde0e2402c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.201340 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9615f0d1bb9d46ba8e855fb5819eb5a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.202109 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 111f5e9afc244ad2bd04df715d85bbe9
I20250109 20:53:08.204102 2475 raft_consensus.cc:491] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.204246 2475 raft_consensus.cc:513] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.204317 447 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:08.204742 1160 ts_tablet_manager.cc:1918] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.204850 1160 log.cc:1198] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/59ebeffe516c4c239ae2ee9040276e53
I20250109 20:53:08.204911 2475 leader_election.cc:290] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:08.204978 1027 tablet_service.cc:1467] Processing CreateTablet for tablet b9e11a8181ae42619a56092017fa9315 (DEFAULT_TABLE table=default.Table_e9289eacd1074dd59e244001042ccbe2 [id=5de4bd9d734647be93e19d743b778ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.205183 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 111f5e9afc244ad2bd04df715d85bbe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.205324 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e11a8181ae42619a56092017fa9315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.205502 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ee36f3be324b14868976298ca26118" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:08.206045 990 leader_election.cc:343] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.206599 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ee36f3be324b14868976298ca26118" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.207001 1176 raft_consensus.cc:738] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:53:08.207131 989 leader_election.cc:343] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.207226 989 leader_election.cc:304] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.207191 1176 consensus_queue.cc:260] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.207993 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 50039da210df47ab8476e28387c8a802 (table default.Table_422152ed650445aeb8f7af679f9c1d12 [id=b296e8173f88419a8a14ac395611b4bc]). Expected version 3 got 0
I20250109 20:53:08.208317 2633 raft_consensus.cc:491] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.208439 2633 raft_consensus.cc:513] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.208946 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d9ef0e5255a74d8596dbacdac71c85ca (DEFAULT_TABLE table=default.Table_9234d242e6e44e3494df97820a63cc88 [id=9e02de64790b4fe8bb393842ba201a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.209038 2633 leader_election.cc:290] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.209002 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_776ff5756e184f24bcd08d48461ec4e5" } new_table_name: "default.Table_f4a8579df694421da36c814ac51cbf73" modify_external_catalogs: true
I20250109 20:53:08.209216 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ef0e5255a74d8596dbacdac71c85ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:08.209982 854 leader_election.cc:343] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 80da78917f0b4931ab1705371c1fe5dd
I20250109 20:53:08.212088 2534 raft_consensus.cc:491] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.212210 2534 raft_consensus.cc:513] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.212761 2534 leader_election.cc:290] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.212918 1025 tablet_service.cc:1467] Processing CreateTablet for tablet eae1da4227e54d17bc98ff68f5c36d4f (DEFAULT_TABLE table=default.Table_e2a32f40c0aa43d8bfb4cc97d297fc76 [id=9f7d9df769fd43e48208451a4ce3718b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.213171 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae1da4227e54d17bc98ff68f5c36d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.213380 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6569b450bb7a4a78922061d2ede981fe" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:08.213785 712 leader_election.cc:343] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.214756 2475 raft_consensus.cc:2743] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:08.215919 713 leader_election.cc:343] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 6569b450bb7a4a78922061d2ede981fe
I20250109 20:53:08.216015 713 leader_election.cc:304] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:08.216363 2534 raft_consensus.cc:2743] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:08.219249 854 leader_election.cc:343] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 4160bb4ba40347bda48f8fa88f8b6001
I20250109 20:53:08.219339 854 leader_election.cc:304] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:08.219383 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e9398f839f4604819e82794af94351. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:08.219676 854 leader_election.cc:343] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 80da78917f0b4931ab1705371c1fe5dd
I20250109 20:53:08.219755 854 leader_election.cc:304] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:08.220006 2632 raft_consensus.cc:2743] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.220129 2632 raft_consensus.cc:2743] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.222595 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 111f5e9afc244ad2bd04df715d85bbe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.222780 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814361a1e854451a0ee7469cc6d447b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.222982 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ee36f3be324b14868976298ca26118 with delete_type TABLET_DATA_DELETED (replaced by tablet 407c57dfd4134eeba07f74386540f844) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.226845 1160 ts_tablet_manager.cc:1939] T 59ebeffe516c4c239ae2ee9040276e53 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.227329 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 59ebeffe516c4c239ae2ee9040276e53 (table default.Table_887acb86017941c4b8636c1008d835d9 [id=facd2e83422c492aa4337beff9ceb548]) successfully deleted
I20250109 20:53:08.227558 1176 ts_tablet_manager.cc:1428] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.120s user 0.015s sys 0.001s
I20250109 20:53:08.232900 1176 tablet_bootstrap.cc:492] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.234071 1176 tablet_bootstrap.cc:654] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:08.239354 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 086c6b621cae4214bcec4daa1a725272 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 086c6b621cae4214bcec4daa1a725272 already in progress: creating tablet
I20250109 20:53:08.239565 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 27cb4e6e3a5944bbadb8f1d619d7361c
I20250109 20:53:08.239912 450 catalog_manager.cc:6596] ReplaceTablet: tablet 27cb4e6e3a5944bbadb8f1d619d7361c deleted and replaced by tablet fc2937e6375b424b901b7ec413ed9648
I20250109 20:53:08.240159 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 27cb4e6e3a5944bbadb8f1d619d7361c with delete_type TABLET_DATA_DELETED (replaced by tablet fc2937e6375b424b901b7ec413ed9648) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.241115 1160 tablet_replica.cc:331] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.241266 1160 raft_consensus.cc:2238] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.241391 1160 raft_consensus.cc:2267] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.241809 1160 ts_tablet_manager.cc:1905] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.242168 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2383a4fe6f864cf68643fe3d59493438"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.242630 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2383a4fe6f864cf68643fe3d59493438 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.243630 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2d3a253685e34e80a26f338be66d31bb
I20250109 20:53:08.244158 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_95fcb9f283734637832d08ce71e8277e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.244541 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95fcb9f283734637832d08ce71e8277e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.239926 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 51b760fcb4fc466ba49d456892d43f3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51b760fcb4fc466ba49d456892d43f3c already in progress: creating tablet
W20250109 20:53:08.245244 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a1ed66fe3b46456ea784a3471575c052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ed66fe3b46456ea784a3471575c052 already in progress: creating tablet
W20250109 20:53:08.245512 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 40546e1b5e854a79a7f706a43b6c6062 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40546e1b5e854a79a7f706a43b6c6062 already in progress: creating tablet
W20250109 20:53:08.245687 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2098d97e88be463c89ad3ea42e26938c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2098d97e88be463c89ad3ea42e26938c already in progress: creating tablet
I20250109 20:53:08.247471 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 866f74d8bf534206b6d1642dbb42b603
I20250109 20:53:08.247786 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5ced7e77dcb4404d95175dab42615cee
I20250109 20:53:08.248019 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_213216398c104dca9ffe9a505d0e46fe" } new_table_name: "default.Table_a0d5d5044e9a452d9d5b8b9529adf615" modify_external_catalogs: true
I20250109 20:53:08.248207 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4f8710d177654a9c81c383e7aac0841b" } new_table_name: "default.Table_f2d8dd684ea6482baa72a97936251de5" modify_external_catalogs: true
I20250109 20:53:08.249150 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b5bcf393703c4c9694190923216585d9
I20250109 20:53:08.249652 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9657151670d0422b879b12781fed0eb5" } new_table_name: "default.Table_c38beb2126e94974a129403bd9109727" modify_external_catalogs: true
I20250109 20:53:08.250830 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bcf393703c4c9694190923216585d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.250909 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d4029d7ef44145458ee919d3c842bfdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.251293 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4029d7ef44145458ee919d3c842bfdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.253197 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ced7e77dcb4404d95175dab42615cee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.253360 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bcf393703c4c9694190923216585d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.255527 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d036d7adca22490991c6e6ca34086cbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.256009 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d036d7adca22490991c6e6ca34086cbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.257994 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 27cb4e6e3a5944bbadb8f1d619d7361c with delete_type TABLET_DATA_DELETED (replaced by tablet fc2937e6375b424b901b7ec413ed9648) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.258412 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3a253685e34e80a26f338be66d31bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.258687 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 866f74d8bf534206b6d1642dbb42b603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.258952 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ced7e77dcb4404d95175dab42615cee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.259150 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3a253685e34e80a26f338be66d31bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.259335 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3a253685e34e80a26f338be66d31bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.259624 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 27cb4e6e3a5944bbadb8f1d619d7361c with delete_type TABLET_DATA_DELETED (replaced by tablet fc2937e6375b424b901b7ec413ed9648) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.259773 746 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bcf393703c4c9694190923216585d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.250839 1160 ts_tablet_manager.cc:1918] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.262897 1160 log.cc:1198] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/72cc0bbbb5584f14ba4d6c62211a12c5
I20250109 20:53:08.263185 1160 ts_tablet_manager.cc:1939] T 72cc0bbbb5584f14ba4d6c62211a12c5 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.263537 1160 tablet_replica.cc:331] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.263645 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 72cc0bbbb5584f14ba4d6c62211a12c5 (table default.Table_4887e41bfb6f4a4aa53cb437224c8e1c [id=6f8c122db03b488393e92bdacc9f5b09]) successfully deleted
I20250109 20:53:08.263691 1160 raft_consensus.cc:2238] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.263806 1160 raft_consensus.cc:2267] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.264245 1160 ts_tablet_manager.cc:1905] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.268957 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 866f74d8bf534206b6d1642dbb42b603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.269376 892 tablet_service.cc:1467] Processing CreateTablet for tablet 30e9398f839f4604819e82794af94351 (DEFAULT_TABLE table=default.Table_7a9f34b8d72242458260fc2cc645f85f [id=397fad389c9b45e7bb0a2017841c7a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.269775 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e9398f839f4604819e82794af94351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.269992 883 tablet_service.cc:1467] Processing CreateTablet for tablet abea7dc7470a4b97a112f4869cff48dd (DEFAULT_TABLE table=default.Table_6ecacbb0b64b49938e20e76653b50f4c [id=a473165ac52149a48be98283ae41fcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.270258 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abea7dc7470a4b97a112f4869cff48dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.271736 1166 ts_tablet_manager.cc:1918] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:08.271836 1166 log.cc:1198] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/bc32ce3dff0f4d18bb17c6784b0bab41
I20250109 20:53:08.272156 1166 ts_tablet_manager.cc:1939] T bc32ce3dff0f4d18bb17c6784b0bab41 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:08.273017 1176 tablet_bootstrap.cc:492] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.273134 1176 ts_tablet_manager.cc:1397] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.003s
I20250109 20:53:08.273321 893 tablet_service.cc:1467] Processing CreateTablet for tablet 751e9f4f46c2481488bddf8b89934dea (DEFAULT_TABLE table=default.Table_a6bfa7b3b5654fe4ad9b3401476d068d [id=3e1ff8cbd9444155962cf71cf38d6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.273636 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 751e9f4f46c2481488bddf8b89934dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.273651 1176 raft_consensus.cc:357] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.273823 1176 raft_consensus.cc:383] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.273864 1176 raft_consensus.cc:738] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.274022 1176 consensus_queue.cc:260] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.274456 1176 ts_tablet_manager.cc:1428] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:08.277699 1176 tablet_bootstrap.cc:492] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.279016 1176 tablet_bootstrap.cc:654] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.280175 895 tablet_service.cc:1467] Processing CreateTablet for tablet 613fef2a18fd483fb58ca66c4c1b346e (DEFAULT_TABLE table=default.Table_d1a9999752c4415ca70dcb514d0e0003 [id=b160b9bc7e574e1896000b46a6114ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.280457 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 613fef2a18fd483fb58ca66c4c1b346e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.282315 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 866f74d8bf534206b6d1642dbb42b603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.283805 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 613fef2a18fd483fb58ca66c4c1b346e (DEFAULT_TABLE table=default.Table_d1a9999752c4415ca70dcb514d0e0003 [id=b160b9bc7e574e1896000b46a6114ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.284118 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 613fef2a18fd483fb58ca66c4c1b346e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.284636 1160 ts_tablet_manager.cc:1918] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.284704 1160 log.cc:1198] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/686ed5f7d5524c54a8debe1bc610bd25
I20250109 20:53:08.284976 1160 ts_tablet_manager.cc:1939] T 686ed5f7d5524c54a8debe1bc610bd25 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.284955 750 tablet_service.cc:1467] Processing CreateTablet for tablet 751e9f4f46c2481488bddf8b89934dea (DEFAULT_TABLE table=default.Table_a6bfa7b3b5654fe4ad9b3401476d068d [id=3e1ff8cbd9444155962cf71cf38d6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.285306 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 751e9f4f46c2481488bddf8b89934dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.288601 757 tablet_service.cc:1467] Processing CreateTablet for tablet e75ed689db244ef9a86c4d87f24dcde9 (DEFAULT_TABLE table=default.Table_f114cef231344fd4a0a4c9e46ae277e5 [id=f15e4b5c5d204ea2875ba6b4fba09329]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.288965 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75ed689db244ef9a86c4d87f24dcde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.288973 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 82c5631eed4749349c20145c3e27a720 (DEFAULT_TABLE table=default.Table_0a8b24b301ef44c686368ec4f0ea1714 [id=aaf137c696144e4a85caf739aadf7eb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.289292 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c5631eed4749349c20145c3e27a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.293058 741 tablet_service.cc:1467] Processing CreateTablet for tablet 613fef2a18fd483fb58ca66c4c1b346e (DEFAULT_TABLE table=default.Table_d1a9999752c4415ca70dcb514d0e0003 [id=b160b9bc7e574e1896000b46a6114ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.293395 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 613fef2a18fd483fb58ca66c4c1b346e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.293807 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 111f5e9afc244ad2bd04df715d85bbe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.293988 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet c70ddcc69b2b42fcbe133978dd3ac05b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:05 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.294275 1022 tablet_service.cc:1467] Processing CreateTablet for tablet abea7dc7470a4b97a112f4869cff48dd (DEFAULT_TABLE table=default.Table_6ecacbb0b64b49938e20e76653b50f4c [id=a473165ac52149a48be98283ae41fcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.294549 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abea7dc7470a4b97a112f4869cff48dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.296813 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ced7e77dcb4404d95175dab42615cee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.297088 749 tablet_service.cc:1467] Processing CreateTablet for tablet 82c5631eed4749349c20145c3e27a720 (DEFAULT_TABLE table=default.Table_0a8b24b301ef44c686368ec4f0ea1714 [id=aaf137c696144e4a85caf739aadf7eb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.297233 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 751e9f4f46c2481488bddf8b89934dea (DEFAULT_TABLE table=default.Table_a6bfa7b3b5654fe4ad9b3401476d068d [id=3e1ff8cbd9444155962cf71cf38d6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.297391 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c5631eed4749349c20145c3e27a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.297577 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 751e9f4f46c2481488bddf8b89934dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.297910 2475 raft_consensus.cc:491] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.298020 2475 raft_consensus.cc:513] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.298637 2475 leader_election.cc:290] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:08.299185 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c717e446858400789dd0e84b4fb60cd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:08.299633 990 leader_election.cc:343] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.300122 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c717e446858400789dd0e84b4fb60cd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:08.300596 989 leader_election.cc:343] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.300683 989 leader_election.cc:304] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.300755 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 686ed5f7d5524c54a8debe1bc610bd25 (table default.Table_6cca94c4ba974861b95038c43ebf833a [id=bf679a467a78433899ead891f63fed13]) successfully deleted
I20250109 20:53:08.300814 746 tablet_service.cc:1467] Processing CreateTablet for tablet abea7dc7470a4b97a112f4869cff48dd (DEFAULT_TABLE table=default.Table_6ecacbb0b64b49938e20e76653b50f4c [id=a473165ac52149a48be98283ae41fcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.301100 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abea7dc7470a4b97a112f4869cff48dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:08.301400 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 94ed3ab88fd8470bbbc27d801323dc9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94ed3ab88fd8470bbbc27d801323dc9d already in progress: creating tablet
I20250109 20:53:08.301548 2475 raft_consensus.cc:2743] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:08.302580 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet dcba62cbadf04761822bf7161c52b758 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcba62cbadf04761822bf7161c52b758 already in progress: creating tablet
I20250109 20:53:08.303984 918 raft_consensus.cc:3054] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:08.308815 1160 tablet_replica.cc:331] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.308969 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 59ce2173203d4dc4910065f830b335ef
I20250109 20:53:08.309043 1160 raft_consensus.cc:2238] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.309425 452 catalog_manager.cc:6596] ReplaceTablet: tablet 59ce2173203d4dc4910065f830b335ef deleted and replaced by tablet e77404f75b22473ba40de0c6854878b7
I20250109 20:53:08.310330 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ce2173203d4dc4910065f830b335ef with delete_type TABLET_DATA_DELETED (replaced by tablet e77404f75b22473ba40de0c6854878b7) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.310352 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ce2173203d4dc4910065f830b335ef with delete_type TABLET_DATA_DELETED (replaced by tablet e77404f75b22473ba40de0c6854878b7) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.311643 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 23b0e904a87e499f812d208228729baf from {username='slave'} at 127.0.0.1:33768
W20250109 20:53:08.315255 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a13a88398c304492b516266c8d076024 consensus state has no leader
I20250109 20:53:08.315567 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d0c4489891a140f1a781056a38ff0723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.316134 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0c4489891a140f1a781056a38ff0723 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.317059 1176 tablet_bootstrap.cc:492] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.317207 1176 ts_tablet_manager.cc:1397] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.001s
W20250109 20:53:08.317382 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2871457499b64529851a4ec12e80838d consensus state has no leader
I20250109 20:53:08.317473 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_350b1b2845b4456994e4e3e61ea7fd0b" } new_table_name: "default.Table_8e8375b44f314cefaa7d0ee8d652ee79" modify_external_catalogs: true
I20250109 20:53:08.317664 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7d251ee94ed84994a8a464de9c5e4224"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:08.317773 1176 raft_consensus.cc:357] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.317924 1176 raft_consensus.cc:383] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.317968 1176 raft_consensus.cc:738] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
W20250109 20:53:08.318019 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d251ee94ed84994a8a464de9c5e4224 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.318133 1176 consensus_queue.cc:260] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.318531 1176 ts_tablet_manager.cc:1428] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:08.318749 1176 tablet_bootstrap.cc:492] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:53:08.318749 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73b1822cd5744b5197d8141fc66539a0 consensus state has no leader
W20250109 20:53:08.319048 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1779840e861f4e7caab371208ae15c40 consensus state has no leader
I20250109 20:53:08.319721 2633 raft_consensus.cc:491] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.319747 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ce2173203d4dc4910065f830b335ef with delete_type TABLET_DATA_DELETED (replaced by tablet e77404f75b22473ba40de0c6854878b7) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.319836 2633 raft_consensus.cc:513] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.319996 1176 tablet_bootstrap.cc:654] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.320466 2633 leader_election.cc:290] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.309203 1160 raft_consensus.cc:2267] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:08.321695 854 leader_election.cc:343] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: b68d23fb2df743f4afa2682f6d2e3efa
W20250109 20:53:08.321863 854 leader_election.cc:343] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: b68d23fb2df743f4afa2682f6d2e3efa
I20250109 20:53:08.321935 854 leader_election.cc:304] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:08.322232 2633 raft_consensus.cc:2743] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.280191 883 tablet_service.cc:1467] Processing CreateTablet for tablet 82c5631eed4749349c20145c3e27a720 (DEFAULT_TABLE table=default.Table_0a8b24b301ef44c686368ec4f0ea1714 [id=aaf137c696144e4a85caf739aadf7eb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.325387 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c5631eed4749349c20145c3e27a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.326007 2620 catalog_manager.cc:1260] Loaded cluster ID: 1852a89fe3204670aafdd57bab642604
I20250109 20:53:08.326138 2620 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: loading cluster ID for follower catalog manager: success
W20250109 20:53:08.330829 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18c0394c53314b708aa12343bcd3e0a9 consensus state has no leader
I20250109 20:53:08.332134 2625 tablet.cc:1721] T 165f6304acc849569a8e90d135a8e2d0 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:08.332675 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 23b0e904a87e499f812d208228729baf
I20250109 20:53:08.334816 452 catalog_manager.cc:6596] ReplaceTablet: tablet 23b0e904a87e499f812d208228729baf deleted and replaced by tablet ad84125cdb2349b0846e3aab797eacdf
I20250109 20:53:08.335364 1160 ts_tablet_manager.cc:1905] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.336285 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8402eec50b7e4f5983d31f8b063ddec9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.336774 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8402eec50b7e4f5983d31f8b063ddec9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.338690 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b0e904a87e499f812d208228729baf with delete_type TABLET_DATA_DELETED (replaced by tablet ad84125cdb2349b0846e3aab797eacdf) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.339329 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b0e904a87e499f812d208228729baf with delete_type TABLET_DATA_DELETED (replaced by tablet ad84125cdb2349b0846e3aab797eacdf) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.339617 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b0e904a87e499f812d208228729baf with delete_type TABLET_DATA_DELETED (replaced by tablet ad84125cdb2349b0846e3aab797eacdf) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.342351 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 94afec96971241cea225c2cf9da1b9c5 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:08.343258 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e75ed689db244ef9a86c4d87f24dcde9 (DEFAULT_TABLE table=default.Table_f114cef231344fd4a0a4c9e46ae277e5 [id=f15e4b5c5d204ea2875ba6b4fba09329]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.343660 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75ed689db244ef9a86c4d87f24dcde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.344044 1176 tablet_bootstrap.cc:492] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.344168 1176 ts_tablet_manager.cc:1397] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.002s
I20250109 20:53:08.344745 1176 raft_consensus.cc:357] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.344905 1176 raft_consensus.cc:383] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.344957 1176 raft_consensus.cc:738] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.345139 1176 consensus_queue.cc:260] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.345695 1176 ts_tablet_manager.cc:1428] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:53:08.345676 895 tablet_service.cc:1467] Processing CreateTablet for tablet e75ed689db244ef9a86c4d87f24dcde9 (DEFAULT_TABLE table=default.Table_f114cef231344fd4a0a4c9e46ae277e5 [id=f15e4b5c5d204ea2875ba6b4fba09329]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.346508 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75ed689db244ef9a86c4d87f24dcde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.347172 1160 ts_tablet_manager.cc:1918] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.347270 1160 log.cc:1198] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ea477dc845614ea78e71e45f862b57ff
I20250109 20:53:08.348968 1160 ts_tablet_manager.cc:1939] T ea477dc845614ea78e71e45f862b57ff P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.349696 1160 tablet_replica.cc:331] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.349795 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ea477dc845614ea78e71e45f862b57ff (table default.Table_1b9aa886edd542fc8039fd921d1048c8 [id=d64f7249495e433fa9c57277ee5eaba9]) successfully deleted
I20250109 20:53:08.349848 1160 raft_consensus.cc:2238] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.349948 1160 raft_consensus.cc:2267] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.350337 1160 ts_tablet_manager.cc:1905] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.363166 1160 ts_tablet_manager.cc:1918] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.363274 1160 log.cc:1198] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/246fd9ac96f54d008e8203e4580e7ef3
I20250109 20:53:08.363555 1160 ts_tablet_manager.cc:1939] T 246fd9ac96f54d008e8203e4580e7ef3 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.364429 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 246fd9ac96f54d008e8203e4580e7ef3 (table default.Table_8ed167dde7b94790b7731c10aa30628c [id=87a978ea0af846f0b3ea56a3d1942d6d]) successfully deleted
I20250109 20:53:08.364796 1160 tablet_replica.cc:331] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.365010 1160 raft_consensus.cc:2238] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.365211 1160 raft_consensus.cc:2267] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.366466 1160 ts_tablet_manager.cc:1905] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:08.369058 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3974c4196d4043be9ad6e0452ed9cb7b (table default.Table_e6daf126f09444e09bde665034161991 [id=16cb6983d685473c8d7aba4e39573aae], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3974c4196d4043be9ad6e0452ed9cb7b consensus state has no leader
I20250109 20:53:08.371790 1160 ts_tablet_manager.cc:1918] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.371894 1160 log.cc:1198] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/8f420b7e623243e093d26c95276c5af9
I20250109 20:53:08.372148 1160 ts_tablet_manager.cc:1939] T 8f420b7e623243e093d26c95276c5af9 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.373126 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 8f420b7e623243e093d26c95276c5af9 (table default.Table_bb47b9b8617f4706aeb895a16ded22ee [id=ec17429d6df548118b1e53e77bac80c0]) successfully deleted
W20250109 20:53:08.373769 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d914848ad47142f4b64597fd00abdacd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d914848ad47142f4b64597fd00abdacd already in progress: creating tablet
I20250109 20:53:08.374084 1160 tablet_replica.cc:331] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.374341 1160 raft_consensus.cc:2238] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.374863 1160 raft_consensus.cc:2267] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.375105 2413 raft_consensus.cc:491] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.375221 2413 raft_consensus.cc:513] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.375757 2413 leader_election.cc:290] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.376883 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c865faed94c3493ea638826bdcd4253d" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:08.377384 712 leader_election.cc:343] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:08.377764 713 leader_election.cc:343] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: c865faed94c3493ea638826bdcd4253d
I20250109 20:53:08.377866 713 leader_election.cc:304] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:08.377943 1160 ts_tablet_manager.cc:1905] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.378340 2413 raft_consensus.cc:2743] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.380108 1176 tablet_bootstrap.cc:492] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.381363 1176 tablet_bootstrap.cc:654] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.391850 1160 ts_tablet_manager.cc:1918] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.391968 1160 log.cc:1198] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4160bb4ba40347bda48f8fa88f8b6001
I20250109 20:53:08.392253 1160 ts_tablet_manager.cc:1939] T 4160bb4ba40347bda48f8fa88f8b6001 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.393162 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4160bb4ba40347bda48f8fa88f8b6001 (table default.Table_13f472ba51244613ab6c1c0cc5e9f9e4 [id=3fc24473de2845628a019824b932a4e6]) successfully deleted
I20250109 20:53:08.394647 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 31de9638669040db9674d3aa7df32103 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.394922 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 374df8c6d1424697a3b3922f777e9870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.395041 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1391) took 1603 ms. Trace:
I20250109 20:53:08.395154 1166 rpcz_store.cc:276] 0109 20:53:06.792025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.793004 (+ 979us) service_pool.cc:225] Handling call
0109 20:53:08.137344 (+1344340us) ts_tablet_manager.cc:1145] Deleting tablet bc32ce3dff0f4d18bb17c6784b0bab41
0109 20:53:08.137354 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.152148 (+ 14794us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.271642 (+119494us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.395019 (+123377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1343974}
I20250109 20:53:08.395524 1166 tablet_replica.cc:331] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:08.395783 1166 raft_consensus.cc:2238] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.395965 1166 raft_consensus.cc:2267] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.396202 1160 tablet_replica.cc:331] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.396447 1166 ts_tablet_manager.cc:1905] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.396631 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet bc32ce3dff0f4d18bb17c6784b0bab41 (table default.Table_26f34d2971ef4d2291e2d02e53d39461 [id=fef90e9d048840acaf567c09af9ac9b8]) successfully deleted
I20250109 20:53:08.396793 1160 raft_consensus.cc:2238] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.396953 1160 raft_consensus.cc:2267] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.397830 1160 ts_tablet_manager.cc:1905] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.400880 1176 tablet_bootstrap.cc:492] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.400993 1176 ts_tablet_manager.cc:1397] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20250109 20:53:08.401628 1176 raft_consensus.cc:357] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.401787 1176 raft_consensus.cc:383] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.401836 1176 raft_consensus.cc:738] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.401985 1176 consensus_queue.cc:260] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.404572 1176 ts_tablet_manager.cc:1428] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:53:08.404839 1176 tablet_bootstrap.cc:492] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.406224 1176 tablet_bootstrap.cc:654] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.407670 1166 ts_tablet_manager.cc:1918] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.407776 1166 log.cc:1198] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/5b2fcddba2ac4d82b2d25bfc64a5967e
I20250109 20:53:08.408025 1166 ts_tablet_manager.cc:1939] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:08.411684 1160 ts_tablet_manager.cc:1918] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.411805 1160 log.cc:1198] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/80da78917f0b4931ab1705371c1fe5dd
I20250109 20:53:08.412111 1160 ts_tablet_manager.cc:1939] T 80da78917f0b4931ab1705371c1fe5dd P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.413132 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 80da78917f0b4931ab1705371c1fe5dd (table default.Table_e6daf126f09444e09bde665034161991 [id=16cb6983d685473c8d7aba4e39573aae]) successfully deleted
I20250109 20:53:08.413585 1160 tablet_replica.cc:331] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.413800 1160 raft_consensus.cc:2238] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.413949 1160 raft_consensus.cc:2267] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.414710 1160 ts_tablet_manager.cc:1905] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.416416 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_75546481297c4d7f9038188240eb1852" } modify_external_catalogs: true
I20250109 20:53:08.416640 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_75546481297c4d7f9038188240eb1852" } modify_external_catalogs: true
I20250109 20:53:08.417443 1176 tablet_bootstrap.cc:492] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.417587 1176 ts_tablet_manager.cc:1397] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250109 20:53:08.418383 1176 raft_consensus.cc:357] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.418587 1176 raft_consensus.cc:383] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.418653 1176 raft_consensus.cc:738] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.418676 2475 raft_consensus.cc:491] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.418784 2475 raft_consensus.cc:513] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.418848 1176 consensus_queue.cc:260] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.419349 2475 leader_election.cc:290] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:08.420109 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de9638669040db9674d3aa7df32103" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.420490 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de9638669040db9674d3aa7df32103" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:08.421520 1157 ts_tablet_manager.cc:1918] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.421638 1157 log.cc:1198] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/dc235614171f4a61b75492ddeb3f0fe6
I20250109 20:53:08.421765 1176 ts_tablet_manager.cc:1428] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250109 20:53:08.421972 1157 ts_tablet_manager.cc:1939] T dc235614171f4a61b75492ddeb3f0fe6 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:53:08.422238 990 leader_election.cc:343] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:08.422729 989 leader_election.cc:343] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.422837 989 leader_election.cc:304] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.425233 1165 tablet_bootstrap.cc:492] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:08.425372 1165 ts_tablet_manager.cc:1397] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.304s user 0.008s sys 0.019s
I20250109 20:53:08.426074 1165 raft_consensus.cc:357] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.426262 1165 raft_consensus.cc:383] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.426311 1165 raft_consensus.cc:738] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.426496 1165 consensus_queue.cc:260] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.427007 1165 ts_tablet_manager.cc:1428] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:08.428213 1165 tablet_bootstrap.cc:492] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:08.429960 1165 tablet_bootstrap.cc:654] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.432729 446 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:08.434383 2534 raft_consensus.cc:491] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.434506 2534 raft_consensus.cc:513] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.435062 2534 leader_election.cc:290] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.436195 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 4e29d5e170b040328770ef8b3a22d462
I20250109 20:53:08.437322 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404ee892ab79427baaef6d19c57699d0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.437412 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e29d5e170b040328770ef8b3a22d462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:08.437798 712 leader_election.cc:343] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.438295 446 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:08.439126 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e29d5e170b040328770ef8b3a22d462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.441001 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e29d5e170b040328770ef8b3a22d462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.445472 1158 tablet_bootstrap.cc:492] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:08.445641 1158 ts_tablet_manager.cc:1397] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.392s user 0.014s sys 0.013s
I20250109 20:53:08.445915 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_875ee088902c4843a6b6a6403666cd51" } modify_external_catalogs: true
I20250109 20:53:08.446136 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_875ee088902c4843a6b6a6403666cd51" } modify_external_catalogs: true
I20250109 20:53:08.448597 2475 raft_consensus.cc:2743] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.449443 1158 raft_consensus.cc:357] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.449610 1158 raft_consensus.cc:383] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.449658 1158 raft_consensus.cc:738] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.449889 1158 consensus_queue.cc:260] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.450393 1158 ts_tablet_manager.cc:1428] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:08.455879 2620 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: acquiring CA information for follower catalog manager: success
I20250109 20:53:08.456573 1158 tablet_bootstrap.cc:492] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:08.458514 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet dc235614171f4a61b75492ddeb3f0fe6 (table default.Table_fc79e5e4f5114b189904f056b147d163 [id=aea1bb0e692345379ba28d2c4c7f3fb8]) successfully deleted
I20250109 20:53:08.460803 1157 tablet_replica.cc:331] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:08.460911 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ad073c54c6aa42579dde0692025efcef" } modify_external_catalogs: true
I20250109 20:53:08.460999 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_502dc59bfc324b38ba72180fd2141f62" } modify_external_catalogs: true
I20250109 20:53:08.461046 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ad073c54c6aa42579dde0692025efcef" } modify_external_catalogs: true
I20250109 20:53:08.461084 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_502dc59bfc324b38ba72180fd2141f62" } modify_external_catalogs: true
I20250109 20:53:08.461326 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a648ce3f75494fc5bfeb8d6da9685375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:08.461464 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4ed44e516e92436080e6b73ac17291c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.461751 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a648ce3f75494fc5bfeb8d6da9685375 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.461813 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ed44e516e92436080e6b73ac17291c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.462129 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ecc48717bbbc4fc1ad0885259f58bc87" } modify_external_catalogs: true
I20250109 20:53:08.462257 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ecc48717bbbc4fc1ad0885259f58bc87" } modify_external_catalogs: true
I20250109 20:53:08.463094 1157 raft_consensus.cc:2238] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.463236 1157 raft_consensus.cc:2267] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.463724 1157 ts_tablet_manager.cc:1905] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.472011 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 365d3a2617114ca19de45765a1e95513
I20250109 20:53:08.475154 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9c2eb4c8de43470a9540162b9cf17db7
I20250109 20:53:08.475802 1157 ts_tablet_manager.cc:1918] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.479327 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c2eb4c8de43470a9540162b9cf17db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.479738 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c2eb4c8de43470a9540162b9cf17db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.479789 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 365d3a2617114ca19de45765a1e95513 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.481592 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7c9a4ad54fef41c3941fd9a02be5a418
I20250109 20:53:08.481637 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d45678706a1f4f549e17acd939c9f6e3
I20250109 20:53:08.482313 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9a4ad54fef41c3941fd9a02be5a418 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.483630 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9a4ad54fef41c3941fd9a02be5a418 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.484261 1160 ts_tablet_manager.cc:1918] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.489497 1176 tablet_bootstrap.cc:492] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.491523 1176 tablet_bootstrap.cc:654] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.496759 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e97ff530df46ae80eb1b5615a55a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.497701 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cdf5b737504d4e8f598cd2b265192e with delete_type TABLET_DATA_DELETED (replaced by tablet 1591374d521a47ebb405a812fe656b19) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.498050 2684 raft_consensus.cc:491] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.498162 2684 raft_consensus.cc:513] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.498692 2684 leader_election.cc:290] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:08.500244 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1388) took 1708 ms. Trace:
I20250109 20:53:08.500330 1166 rpcz_store.cc:276] 0109 20:53:06.791702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.796801 (+ 5099us) service_pool.cc:225] Handling call
0109 20:53:08.395471 (+1598670us) ts_tablet_manager.cc:1145] Deleting tablet 5b2fcddba2ac4d82b2d25bfc64a5967e
0109 20:53:08.395483 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.399513 (+ 4030us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.407599 (+ 8086us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.500227 (+ 92628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598461}
I20250109 20:53:08.500687 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 5b2fcddba2ac4d82b2d25bfc64a5967e (table default.Table_7f957a53ae794811bf294551bf3f6e00 [id=8097eb217f1844ec8e2dc236b2f19256]) successfully deleted
I20250109 20:53:08.479185 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c2eb4c8de43470a9540162b9cf17db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.479671 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 365d3a2617114ca19de45765a1e95513 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.483533 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9a4ad54fef41c3941fd9a02be5a418 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.486222 1158 tablet_bootstrap.cc:654] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.502636 1157 log.cc:1198] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/404ee892ab79427baaef6d19c57699d0
I20250109 20:53:08.503175 1157 ts_tablet_manager.cc:1939] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:08.507333 2687 raft_consensus.cc:491] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.507798 2687 raft_consensus.cc:513] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.496868 2534 raft_consensus.cc:491] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.508095 2534 raft_consensus.cc:513] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.508608 2687 leader_election.cc:290] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:08.507673 1166 tablet_replica.cc:331] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:53:08.489208 713 leader_election.cc:343] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 404ee892ab79427baaef6d19c57699d0
I20250109 20:53:08.508684 2534 leader_election.cc:290] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.508903 1166 raft_consensus.cc:2238] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.509320 1166 raft_consensus.cc:2267] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.509074 713 leader_election.cc:304] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:08.509457 713 raft_consensus.cc:2681] T 404ee892ab79427baaef6d19c57699d0 P 09833ea46b7141d58c3899dace69e49f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250109 20:53:08.509786 1166 ts_tablet_manager.cc:1905] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.511046 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3911b54a040144baadfc4e0088d58e64" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:08.511240 1055 raft_consensus.cc:2463] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:08.511875 713 leader_election.cc:304] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:08.512162 2534 raft_consensus.cc:2798] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:08.512229 2534 raft_consensus.cc:491] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:08.512282 2534 raft_consensus.cc:3054] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:08.516363 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet e457baa02f4e48f8a627ffdb05267fc8 (table default.Table_a4b39fd34a0e444cb84557a81907e39e [id=c7ac68af36ab444ea36032ce72130972]). Expected version 1 got 0
I20250109 20:53:08.517736 2534 raft_consensus.cc:513] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.518780 1160 log.cc:1198] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/e090f72eb1e046319b825ee5fe126332
I20250109 20:53:08.499126 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e97ff530df46ae80eb1b5615a55a3f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.509034 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70ddcc69b2b42fcbe133978dd3ac05b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.509152 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3911b54a040144baadfc4e0088d58e64" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.484817 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 365d3a2617114ca19de45765a1e95513 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.521008 1160 ts_tablet_manager.cc:1939] T e090f72eb1e046319b825ee5fe126332 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.521680 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet e090f72eb1e046319b825ee5fe126332 (table default.Table_e9297199b4064b5f889530ac2757c3ad [id=53929cac72e44474a2dbc7ae26524da3]) successfully deleted
I20250109 20:53:08.522104 1160 tablet_replica.cc:331] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.522347 1160 raft_consensus.cc:2238] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.522559 1160 raft_consensus.cc:2267] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.523285 1160 ts_tablet_manager.cc:1905] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.523351 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_826fa37db91b4f23ae2bc375badf74f4" } new_table_name: "default.Table_9b3d177d6e8d4b7c88e6bf73c71a55c0" modify_external_catalogs: true
I20250109 20:53:08.523290 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bc2073b805334d97bd4e5d2afaf1e0ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:08.523663 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c6cc9ff227164e74bf9fbc9e799dd5cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.523787 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc2073b805334d97bd4e5d2afaf1e0ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.524003 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6cc9ff227164e74bf9fbc9e799dd5cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.524430 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_93ab49fd9374420f80e0422073c909da" } new_table_name: "default.Table_26a3d03031054e23aaf1c0591d562501" modify_external_catalogs: true
I20250109 20:53:08.524024 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f54d04dc297c4ff39b8ad462455c0deb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.525220 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f54d04dc297c4ff39b8ad462455c0deb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.525305 989 leader_election.cc:343] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:08.525512 989 leader_election.cc:343] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.526213 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a6bfa7b3b5654fe4ad9b3401476d068d" } new_table_name: "default.Table_869318a5611d42fcbe674f19726997ae" modify_external_catalogs: true
I20250109 20:53:08.520311 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e97ff530df46ae80eb1b5615a55a3f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:08.521931 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70ddcc69b2b42fcbe133978dd3ac05b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:08.522576 712 leader_election.cc:343] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.527288 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f41293cf3af5440199ff09445139004d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.527393 990 leader_election.cc:343] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.527482 990 leader_election.cc:304] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:08.527642 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f41293cf3af5440199ff09445139004d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.527951 990 leader_election.cc:343] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.528028 990 leader_election.cc:304] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.528780 2475 raft_consensus.cc:2743] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.529006 2475 raft_consensus.cc:2743] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.528672 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_62fca2bdab804af8ab92b2eb60115336" } new_table_name: "default.Table_567d3232cbbd4fc0a087d039f51d5144" modify_external_catalogs: true
I20250109 20:53:08.532219 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_78986fd7b4744986aaa94a014f7cc06a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.532964 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78986fd7b4744986aaa94a014f7cc06a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.533290 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_44d9cfca7ff645bb8aad32946b74f286" } new_table_name: "default.Table_3b53ab4c12a148e7abed16681ac19d60" modify_external_catalogs: true
I20250109 20:53:08.528437 2534 leader_election.cc:290] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.534682 1165 tablet_bootstrap.cc:492] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:08.534830 1165 ts_tablet_manager.cc:1397] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.107s user 0.007s sys 0.001s
I20250109 20:53:08.535455 1165 raft_consensus.cc:357] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.535705 1165 raft_consensus.cc:383] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.535786 1165 raft_consensus.cc:738] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.535990 1165 consensus_queue.cc:260] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.536474 1165 ts_tablet_manager.cc:1428] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250109 20:53:08.537084 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1e75afbcc4e43f4870bd4ae466e6830 consensus state has no leader
I20250109 20:53:08.542490 1165 tablet_bootstrap.cc:492] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:08.544409 2684 raft_consensus.cc:491] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.544533 2684 raft_consensus.cc:513] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.545177 2684 leader_election.cc:290] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:08.545934 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2769e5906c604bd5b14df79db765ec61" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.547042 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3911b54a040144baadfc4e0088d58e64" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:08.547184 1055 raft_consensus.cc:3054] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:08.547876 989 leader_election.cc:343] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.549541 1055 raft_consensus.cc:2463] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:08.549927 713 leader_election.cc:304] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:08.551437 1160 ts_tablet_manager.cc:1918] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.555927 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_325a0939aec24e8991002b08e86afa13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.556440 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_325a0939aec24e8991002b08e86afa13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.556655 1158 tablet_bootstrap.cc:492] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:08.557315 1176 tablet_bootstrap.cc:492] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.557471 1176 ts_tablet_manager.cc:1397] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.004s
I20250109 20:53:08.557616 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_631065e711734e5d9e89c9ee8f3e3d06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.558027 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_631065e711734e5d9e89c9ee8f3e3d06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.558344 1176 raft_consensus.cc:357] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.558562 1176 raft_consensus.cc:383] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.558651 1176 raft_consensus.cc:738] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.558923 1176 consensus_queue.cc:260] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.559450 1176 ts_tablet_manager.cc:1428] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:08.566856 1176 tablet_bootstrap.cc:492] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:53:08.567296 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df646addd3434934a62702e9ca9731c0 consensus state has no leader
I20250109 20:53:08.568497 1176 tablet_bootstrap.cc:654] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.557407 2534 raft_consensus.cc:491] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.569463 2534 raft_consensus.cc:513] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.570065 2534 leader_election.cc:290] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.570571 1158 ts_tablet_manager.cc:1397] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.000s
I20250109 20:53:08.573405 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet b56d2c0706c84195acceb97ef4985c14 with delete_type TABLET_DATA_DELETED (replaced by tablet 34e3f9d76d1f48ec8e5eebdd4b43ddc2) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.583635 1166 ts_tablet_manager.cc:1918] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.583738 1166 log.cc:1198] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f2c83e709a8d4e078e67af58ec51beac
I20250109 20:53:08.584030 1166 ts_tablet_manager.cc:1939] T f2c83e709a8d4e078e67af58ec51beac P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:08.589581 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_819fde99126d4beb8811b3032f6450bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.590149 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_819fde99126d4beb8811b3032f6450bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.592500 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 751e9f4f46c2481488bddf8b89934dea consensus state has no leader
W20250109 20:53:08.594547 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e53966daf1ed4a67801d9ab239e17d8e consensus state has no leader
W20250109 20:53:08.594838 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af6d061fd1e34117bd8ba609c0638d48 consensus state has no leader
I20250109 20:53:08.600458 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1389) took 1808 ms. Trace:
I20250109 20:53:08.605422 1166 rpcz_store.cc:276] 0109 20:53:06.791803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.797036 (+ 5233us) service_pool.cc:225] Handling call
0109 20:53:08.500946 (+1703910us) ts_tablet_manager.cc:1145] Deleting tablet f2c83e709a8d4e078e67af58ec51beac
0109 20:53:08.507630 (+ 6684us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.550813 (+ 43183us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.583543 (+ 32730us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.600439 (+ 16896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1703689}
I20250109 20:53:08.606161 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1392) took 1732 ms. Trace:
I20250109 20:53:08.606220 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f2c83e709a8d4e078e67af58ec51beac (table default.Table_a7a796c2d1d047baa518202a6b0091b7 [id=4943d76200b14b51b15563815954e38e]) successfully deleted
I20250109 20:53:08.606320 1166 rpcz_store.cc:276] 0109 20:53:06.873753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.874073 (+ 320us) service_pool.cc:225] Handling call
0109 20:53:08.606027 (+1731954us) ts_tablet_manager.cc:1145] Deleting tablet 4597aa225a374487b13647961b1843d8
0109 20:53:08.606039 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.606144 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1731773}
I20250109 20:53:08.606709 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1393) took 1731 ms. Trace:
W20250109 20:53:08.606775 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4597aa225a374487b13647961b1843d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4597aa225a374487b13647961b1843d8 already in progress: creating tablet
I20250109 20:53:08.606791 1166 rpcz_store.cc:276] 0109 20:53:06.875324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:06.875359 (+ 35us) service_pool.cc:225] Handling call
0109 20:53:08.606602 (+1731243us) ts_tablet_manager.cc:1145] Deleting tablet 6cb88771d2b148bdae6e427c1b25b9a9
0109 20:53:08.606613 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.606694 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1731096}
I20250109 20:53:08.607126 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1403) took 1467 ms. Trace:
W20250109 20:53:08.607199 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6cb88771d2b148bdae6e427c1b25b9a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cb88771d2b148bdae6e427c1b25b9a9 already in progress: creating tablet
I20250109 20:53:08.607199 1166 rpcz_store.cc:276] 0109 20:53:07.139661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.177131 (+ 37470us) service_pool.cc:225] Handling call
0109 20:53:08.607038 (+1429907us) ts_tablet_manager.cc:1145] Deleting tablet f2d468467478415ba582be6d4a9d989b
0109 20:53:08.607047 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.607113 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429697}
I20250109 20:53:08.607466 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1404) took 1448 ms. Trace:
W20250109 20:53:08.607551 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f2d468467478415ba582be6d4a9d989b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d468467478415ba582be6d4a9d989b already in progress: creating tablet
I20250109 20:53:08.607549 1166 rpcz_store.cc:276] 0109 20:53:07.159001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.178599 (+ 19598us) service_pool.cc:225] Handling call
0109 20:53:08.607379 (+1428780us) ts_tablet_manager.cc:1145] Deleting tablet 0f7e3652017946d481090b9940d32e97
0109 20:53:08.607387 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.607451 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428617}
W20250109 20:53:08.607957 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 0f7e3652017946d481090b9940d32e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f7e3652017946d481090b9940d32e97 already in progress: creating tablet
I20250109 20:53:08.583911 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2769e5906c604bd5b14df79db765ec61" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:08.584376 713 leader_election.cc:343] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: e716270ffcf44039b075023fd0c2d2a3
W20250109 20:53:08.611541 990 leader_election.cc:343] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.611644 990 leader_election.cc:304] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.611824 2684 raft_consensus.cc:491] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.611932 2684 raft_consensus.cc:513] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.612234 1158 raft_consensus.cc:357] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.612491 2684 leader_election.cc:290] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:08.612607 2684 raft_consensus.cc:2743] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.612797 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1402) took 1473 ms. Trace:
I20250109 20:53:08.612870 1166 rpcz_store.cc:276] 0109 20:53:07.139556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.185570 (+ 46014us) service_pool.cc:225] Handling call
0109 20:53:08.607704 (+1422134us) ts_tablet_manager.cc:1145] Deleting tablet b0dd5ca642034441a0eb5c5a653b7263
0109 20:53:08.607711 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.612786 (+ 5075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1421894}
I20250109 20:53:08.612907 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cdf5b737504d4e8f598cd2b265192e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:08.613145 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1413) took 1283 ms. Trace:
I20250109 20:53:08.613198 1166 rpcz_store.cc:276] 0109 20:53:07.329400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.329449 (+ 49us) service_pool.cc:225] Handling call
0109 20:53:08.613061 (+1283612us) ts_tablet_manager.cc:1145] Deleting tablet 086c6b621cae4214bcec4daa1a725272
0109 20:53:08.613068 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.613132 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1283440}
W20250109 20:53:08.613252 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b0dd5ca642034441a0eb5c5a653b7263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0dd5ca642034441a0eb5c5a653b7263 already in progress: creating tablet
I20250109 20:53:08.613463 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1412) took 1284 ms. Trace:
I20250109 20:53:08.613482 1158 raft_consensus.cc:383] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250109 20:53:08.613569 990 leader_election.cc:343] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.613619 1158 raft_consensus.cc:738] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:08.613845 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 086c6b621cae4214bcec4daa1a725272 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 086c6b621cae4214bcec4daa1a725272 already in progress: creating tablet
I20250109 20:53:08.613819 1158 consensus_queue.cc:260] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.615121 1176 tablet_bootstrap.cc:492] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.615353 1176 ts_tablet_manager.cc:1397] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.003s
I20250109 20:53:08.616034 1176 raft_consensus.cc:357] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.616220 1176 raft_consensus.cc:383] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.616284 1176 raft_consensus.cc:738] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.616484 1176 consensus_queue.cc:260] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.617543 1176 ts_tablet_manager.cc:1428] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:53:08.618729 1176 tablet_bootstrap.cc:492] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.613536 1166 rpcz_store.cc:276] 0109 20:53:07.329050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.332884 (+ 3834us) service_pool.cc:225] Handling call
0109 20:53:08.613379 (+1280495us) ts_tablet_manager.cc:1145] Deleting tablet 94afec96971241cea225c2cf9da1b9c5
0109 20:53:08.613388 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.613450 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280343}
W20250109 20:53:08.621207 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 94afec96971241cea225c2cf9da1b9c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94afec96971241cea225c2cf9da1b9c5 already in progress: creating tablet
I20250109 20:53:08.623183 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1414) took 1280 ms. Trace:
I20250109 20:53:08.623265 1166 rpcz_store.cc:276] 0109 20:53:07.342916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.345295 (+ 2379us) service_pool.cc:225] Handling call
0109 20:53:08.620991 (+1275696us) ts_tablet_manager.cc:1145] Deleting tablet 51b760fcb4fc466ba49d456892d43f3c
0109 20:53:08.623070 (+ 2079us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.623166 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275537}
W20250109 20:53:08.623678 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 51b760fcb4fc466ba49d456892d43f3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51b760fcb4fc466ba49d456892d43f3c already in progress: creating tablet
I20250109 20:53:08.629532 2697 raft_consensus.cc:2798] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:08.630160 1166 tablet_replica.cc:331] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:08.630332 1166 raft_consensus.cc:2238] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.630455 1166 raft_consensus.cc:2267] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.630887 1166 ts_tablet_manager.cc:1905] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.633080 1176 tablet_bootstrap.cc:654] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.634447 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet beca5f1ed9d3434fb7de90cd74a6f09b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.636263 1166 ts_tablet_manager.cc:1918] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.636390 1166 log.cc:1198] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/527776a5d742446eb8fe61c3c855021f
I20250109 20:53:08.638129 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a4b4139a715c423ea5d28aa9c087d962" } modify_external_catalogs: true
I20250109 20:53:08.638298 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a4b4139a715c423ea5d28aa9c087d962" } modify_external_catalogs: true
I20250109 20:53:08.640780 1158 ts_tablet_manager.cc:1428] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20250109 20:53:08.647262 1158 tablet_bootstrap.cc:492] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:08.649216 1176 tablet_bootstrap.cc:492] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.649336 1176 ts_tablet_manager.cc:1397] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.004s
I20250109 20:53:08.650264 1157 tablet_replica.cc:331] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:08.650161 1176 raft_consensus.cc:357] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.650341 1176 raft_consensus.cc:383] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.650388 1176 raft_consensus.cc:738] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.650573 1166 ts_tablet_manager.cc:1939] T 527776a5d742446eb8fe61c3c855021f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:08.650607 1176 consensus_queue.cc:260] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.651089 1176 ts_tablet_manager.cc:1428] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:08.651276 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1415) took 1246 ms. Trace:
I20250109 20:53:08.654939 1158 tablet_bootstrap.cc:654] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.655263 1160 log.cc:1198] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/b68d23fb2df743f4afa2682f6d2e3efa
I20250109 20:53:08.558955 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8539cefebb844b3098a308ca580b0e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.612902 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cdf5b737504d4e8f598cd2b265192e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.657101 1160 ts_tablet_manager.cc:1939] T b68d23fb2df743f4afa2682f6d2e3efa P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.554309 1165 tablet_bootstrap.cc:654] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.661760 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 404ee892ab79427baaef6d19c57699d0 (table default.Table_3abc7e662927447b90d40c4c4ba6e04b [id=d394131690ff4ebaa3d9639758b62f04]) successfully deleted
W20250109 20:53:08.662052 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 94ed3ab88fd8470bbbc27d801323dc9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94ed3ab88fd8470bbbc27d801323dc9d already in progress: creating tablet
W20250109 20:53:08.657349 989 leader_election.cc:343] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.663384 989 leader_election.cc:304] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.663935 2687 raft_consensus.cc:2743] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.664137 1166 rpcz_store.cc:276] 0109 20:53:07.404906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.405166 (+ 260us) service_pool.cc:225] Handling call
0109 20:53:08.623471 (+1218305us) ts_tablet_manager.cc:1145] Deleting tablet 527776a5d742446eb8fe61c3c855021f
0109 20:53:08.630121 (+ 6650us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.633514 (+ 3393us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.636169 (+ 2655us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.651259 (+ 15090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1218100}
I20250109 20:53:08.664598 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 527776a5d742446eb8fe61c3c855021f (table default.Table_748cfdfa67a04dc8bf878fccb596b642 [id=c82368e2d31f4654803844337d1d41a8]) successfully deleted
I20250109 20:53:08.664986 1176 tablet_bootstrap.cc:492] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.666504 1176 tablet_bootstrap.cc:654] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.670105 1166 tablet_replica.cc:331] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:08.670282 1166 raft_consensus.cc:2238] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.670416 1166 raft_consensus.cc:2267] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.670841 1166 ts_tablet_manager.cc:1905] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.679553 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7598437489fd4408a5c7fafb1a722d66" } modify_external_catalogs: true
I20250109 20:53:08.679693 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7598437489fd4408a5c7fafb1a722d66" } modify_external_catalogs: true
I20250109 20:53:08.680366 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e556f33cc6824d0cb0f84e33ae69638e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.680893 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e556f33cc6824d0cb0f84e33ae69638e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.681463 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d13168ca13c400a8a2d9608293ff8c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.681983 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1e5cf45918b64420a95b67afccc51b67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.682408 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e5cf45918b64420a95b67afccc51b67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.683353 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1c1f3e1543534694a3ccf10b8bf07284" } modify_external_catalogs: true
I20250109 20:53:08.683465 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1c1f3e1543534694a3ccf10b8bf07284" } modify_external_catalogs: true
I20250109 20:53:08.683849 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e0cf5166142546989eb825944cae0691" } modify_external_catalogs: true
I20250109 20:53:08.683974 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e0cf5166142546989eb825944cae0691" } modify_external_catalogs: true
I20250109 20:53:08.688400 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b28d493028f4488394c6d101b7b917e6" } modify_external_catalogs: true
I20250109 20:53:08.688530 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b28d493028f4488394c6d101b7b917e6" } modify_external_catalogs: true
I20250109 20:53:08.697875 1166 ts_tablet_manager.cc:1918] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.697988 1166 log.cc:1198] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/3008c672bc544000b24b75ae366d9fb9
I20250109 20:53:08.702560 1166 ts_tablet_manager.cc:1939] T 3008c672bc544000b24b75ae366d9fb9 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:08.703316 1176 tablet_bootstrap.cc:492] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.703505 1176 ts_tablet_manager.cc:1397] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250109 20:53:08.704344 1176 raft_consensus.cc:357] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.704572 1176 raft_consensus.cc:383] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.704653 1176 raft_consensus.cc:738] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.703321 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1416) took 1298 ms. Trace:
I20250109 20:53:08.704906 1176 consensus_queue.cc:260] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.661509 1157 raft_consensus.cc:2238] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.705468 1176 ts_tablet_manager.cc:1428] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:08.705528 1157 raft_consensus.cc:2267] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.705740 1176 tablet_bootstrap.cc:492] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.706080 1157 ts_tablet_manager.cc:1905] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.706955 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet b68d23fb2df743f4afa2682f6d2e3efa (table default.Table_07f19c8f31bb48c598a7a2b8dc7187af [id=e0c7ea945a0a40e0a4a3e369c7455eca]) successfully deleted
I20250109 20:53:08.707263 1160 tablet_replica.cc:331] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.707442 1160 raft_consensus.cc:2238] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.707536 1176 tablet_bootstrap.cc:654] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.707603 1160 raft_consensus.cc:2267] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.708552 1160 ts_tablet_manager.cc:1905] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.661989 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4b7e008a474978b91426f63dbd96ee) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.662835 2697 raft_consensus.cc:695] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:08.709226 2697 consensus_queue.cc:237] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:08.700948 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 172737us
I20250109 20:53:08.711256 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e716270ffcf44039b075023fd0c2d2a3" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.711400 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3911b54a040144baadfc4e0088d58e64" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:08.712299 712 leader_election.cc:343] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.712401 712 leader_election.cc:304] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:08.712742 712 leader_election.cc:343] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.713376 1165 tablet_bootstrap.cc:492] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:08.713500 1165 ts_tablet_manager.cc:1397] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.171s user 0.001s sys 0.004s
I20250109 20:53:08.713775 2534 raft_consensus.cc:2743] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.714521 1165 raft_consensus.cc:357] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.714704 1165 raft_consensus.cc:383] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.714766 1165 raft_consensus.cc:738] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.714984 1165 consensus_queue.cc:260] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.715432 1165 ts_tablet_manager.cc:1428] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:53:08.718302 1157 ts_tablet_manager.cc:1918] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.720784 1157 log.cc:1198] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/c865faed94c3493ea638826bdcd4253d
I20250109 20:53:08.721216 1157 ts_tablet_manager.cc:1939] T c865faed94c3493ea638826bdcd4253d P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:08.722477 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet c865faed94c3493ea638826bdcd4253d (table default.Table_c07faa9c0a64412588c149655620702d [id=be7028b0c99c4883aded712bdd660cbb]) successfully deleted
W20250109 20:53:08.723845 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet dcba62cbadf04761822bf7161c52b758 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcba62cbadf04761822bf7161c52b758 already in progress: creating tablet
I20250109 20:53:08.724364 1157 tablet_replica.cc:331] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:08.724807 1157 raft_consensus.cc:2238] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.704928 1166 rpcz_store.cc:276] 0109 20:53:07.405093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.405890 (+ 797us) service_pool.cc:225] Handling call
0109 20:53:08.664490 (+1258600us) ts_tablet_manager.cc:1145] Deleting tablet 3008c672bc544000b24b75ae366d9fb9
0109 20:53:08.670068 (+ 5578us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.695066 (+ 24998us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.697784 (+ 2718us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.703307 (+ 5523us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258422}
I20250109 20:53:08.725651 1157 raft_consensus.cc:2267] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.726003 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 3008c672bc544000b24b75ae366d9fb9 (table default.Table_47acbd3655b34a70bdefe7ab105d58a1 [id=9917ed9319044ecd84798c91f749eb9f]) successfully deleted
I20250109 20:53:08.726361 1166 tablet_replica.cc:331] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:08.726977 1157 ts_tablet_manager.cc:1905] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.727394 1166 raft_consensus.cc:2238] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.727864 1166 raft_consensus.cc:2267] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.728499 1166 ts_tablet_manager.cc:1905] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.744625 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_78bb824af7494f76a74431aea6b022ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.745154 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78bb824af7494f76a74431aea6b022ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.746300 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e425a9a8d0f49acab65342112e3e64c" } new_table_name: "default.Table_0ac75ea834e740efa59cffbf81f74cb1" modify_external_catalogs: true
I20250109 20:53:08.754169 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ecf49f821ca248e2a9e2e642aed433b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.754732 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecf49f821ca248e2a9e2e642aed433b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.754858 2687 raft_consensus.cc:491] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.754989 2687 raft_consensus.cc:513] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.755539 2687 leader_election.cc:290] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:53:08.756012 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6daac072067b4071bf2280f05c47f14b consensus state has no leader
I20250109 20:53:08.756285 2620 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250109 20:53:08.760038 895 tablet_service.cc:1467] Processing CreateTablet for tablet 49f52fc41bd849dda5584eb6d226c4dc (DEFAULT_TABLE table=default.Table_7d251ee94ed84994a8a464de9c5e4224 [id=a0accb66ece443b2b2c33604a3f1db3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.760622 1165 tablet_bootstrap.cc:492] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:08.761876 1165 tablet_bootstrap.cc:654] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.762115 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9170b3a6b9bb446b8a474ae4df249409" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.763144 2413 raft_consensus.cc:491] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.763270 2413 raft_consensus.cc:513] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.763830 2413 leader_election.cc:290] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:08.765237 989 leader_election.cc:343] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:08.765501 713 leader_election.cc:343] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 4814361a1e854451a0ee7469cc6d447b
I20250109 20:53:08.766124 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4814361a1e854451a0ee7469cc6d447b" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:08.766732 712 leader_election.cc:343] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.766829 712 leader_election.cc:304] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:08.767261 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f52fc41bd849dda5584eb6d226c4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.768353 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a46882e637824d139ac073686c16e46f
I20250109 20:53:08.769065 893 tablet_service.cc:1514] Processing DeleteTablet for tablet a46882e637824d139ac073686c16e46f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.769547 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet a46882e637824d139ac073686c16e46f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.769665 750 tablet_service.cc:1514] Processing DeleteTablet for tablet a46882e637824d139ac073686c16e46f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.770498 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 420f755e81eb4ddd8b872bb7dcea4d13
I20250109 20:53:08.771317 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 420f755e81eb4ddd8b872bb7dcea4d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.771586 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_94097e767754490fa0d9c7402cc172dd" } new_table_name: "default.Table_452790ac382d43de97d706e1328a1543" modify_external_catalogs: true
I20250109 20:53:08.771867 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cbc411f1dddc414ca1c4045adc085f38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.772289 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbc411f1dddc414ca1c4045adc085f38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.774693 2413 raft_consensus.cc:2743] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.778122 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet ade6b4ea4ca44f3eb88d3911b0bbb70b
I20250109 20:53:08.778182 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet edfcbc74f489406297bfa8df228d2787
I20250109 20:53:08.778762 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet dfc973162d464694a4d59423182584ce
I20250109 20:53:08.779250 893 tablet_service.cc:1514] Processing DeleteTablet for tablet edfcbc74f489406297bfa8df228d2787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.779284 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.779558 892 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc973162d464694a4d59423182584ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.781975 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.782148 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 420f755e81eb4ddd8b872bb7dcea4d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.782207 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc973162d464694a4d59423182584ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.782318 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet edfcbc74f489406297bfa8df228d2787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.782776 757 tablet_service.cc:1514] Processing DeleteTablet for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.782964 746 tablet_service.cc:1514] Processing DeleteTablet for tablet edfcbc74f489406297bfa8df228d2787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.783123 749 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc973162d464694a4d59423182584ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.783138 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3af2a00ab04b41915cffc3eb9206bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.783252 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 420f755e81eb4ddd8b872bb7dcea4d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:08.790946 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7db9bca84dfb43f3aca6f12009c4007e consensus state has no leader
I20250109 20:53:08.791577 1158 tablet_bootstrap.cc:492] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:08.791743 1158 ts_tablet_manager.cc:1397] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.145s user 0.003s sys 0.000s
I20250109 20:53:08.792476 1158 raft_consensus.cc:357] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.792661 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9170b3a6b9bb446b8a474ae4df249409" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:08.793031 1158 raft_consensus.cc:383] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.793190 1158 raft_consensus.cc:738] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:08.793326 990 leader_election.cc:343] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.793478 990 leader_election.cc:304] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.793493 1158 consensus_queue.cc:260] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.793826 2687 raft_consensus.cc:2743] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.794354 1158 ts_tablet_manager.cc:1428] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:08.794301 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8335c1b94e274c0d8d8e927554aa0618" } new_table_name: "default.Table_05ab5bd90c8d4f3e97a59539dbeffdc8" modify_external_catalogs: true
I20250109 20:53:08.794582 1158 tablet_bootstrap.cc:492] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:08.795770 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d4c1d0cf8d7f48bfbe31ec25d79cd5d2" } new_table_name: "default.Table_31ad1512dbef48a5b5c8fdfd5443c5f7" modify_external_catalogs: true
I20250109 20:53:08.795132 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a0308450aa42436a8a6535ffdeb4a017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:08.796897 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dc20220cc040480385a78533a6fe34e8" } new_table_name: "default.Table_b53f1d00143b425a941127015bda726a" modify_external_catalogs: true
I20250109 20:53:08.796834 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_74a13a5b1c0e43b99921b064b8c4a82c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.797335 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74a13a5b1c0e43b99921b064b8c4a82c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.797421 1158 tablet_bootstrap.cc:654] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:08.796840 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0308450aa42436a8a6535ffdeb4a017 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.799544 750 tablet_service.cc:1467] Processing CreateTablet for tablet 49f52fc41bd849dda5584eb6d226c4dc (DEFAULT_TABLE table=default.Table_7d251ee94ed84994a8a464de9c5e4224 [id=a0accb66ece443b2b2c33604a3f1db3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.799870 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f52fc41bd849dda5584eb6d226c4dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:08.809516 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de57e94d00d748dc8b02d60f2f1fb27a consensus state has no leader
W20250109 20:53:08.809906 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0075c64fe3164e5688ab5f2c13c736e6 consensus state has no leader
I20250109 20:53:08.811718 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_01a8703293504b22beea98b21800c92e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.812153 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01a8703293504b22beea98b21800c92e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.813266 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5d073a5c6ed8485ea6db09fa96c95579"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.813623 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d073a5c6ed8485ea6db09fa96c95579 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.816154 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bca6602ca0534bf8ab4b47d6e287ee2e consensus state has no leader
I20250109 20:53:08.820942 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 49f52fc41bd849dda5584eb6d226c4dc (DEFAULT_TABLE table=default.Table_7d251ee94ed84994a8a464de9c5e4224 [id=a0accb66ece443b2b2c33604a3f1db3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.820968 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f90c364ce844c9ea8eccb4ae23f8dce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.821332 1021 tablet_service.cc:1467] Processing CreateTablet for tablet aaa1c3ae1e5942febeb58f232f50d052 (DEFAULT_TABLE table=default.Table_d0c4489891a140f1a781056a38ff0723 [id=1457cb332e9148e8af3bc637c5ae9e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.821321 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f52fc41bd849dda5584eb6d226c4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.822477 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e9fda031a43448df8b7eb4840379501a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:08.822799 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaa1c3ae1e5942febeb58f232f50d052. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:08.822919 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9fda031a43448df8b7eb4840379501a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.824117 895 tablet_service.cc:1467] Processing CreateTablet for tablet aaa1c3ae1e5942febeb58f232f50d052 (DEFAULT_TABLE table=default.Table_d0c4489891a140f1a781056a38ff0723 [id=1457cb332e9148e8af3bc637c5ae9e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.824488 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaa1c3ae1e5942febeb58f232f50d052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.824826 750 tablet_service.cc:1467] Processing CreateTablet for tablet aaa1c3ae1e5942febeb58f232f50d052 (DEFAULT_TABLE table=default.Table_d0c4489891a140f1a781056a38ff0723 [id=1457cb332e9148e8af3bc637c5ae9e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.825263 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaa1c3ae1e5942febeb58f232f50d052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.827389 449 catalog_manager.cc:5526] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:08.830600 749 tablet_service.cc:1467] Processing CreateTablet for tablet a07437d4a5a940eb88373c1ee78723cb (DEFAULT_TABLE table=default.Table_d036d7adca22490991c6e6ca34086cbd [id=f0cdf7c1ef7c47b19956c06f5f61d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.830636 892 tablet_service.cc:1467] Processing CreateTablet for tablet a07437d4a5a940eb88373c1ee78723cb (DEFAULT_TABLE table=default.Table_d036d7adca22490991c6e6ca34086cbd [id=f0cdf7c1ef7c47b19956c06f5f61d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.830935 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07437d4a5a940eb88373c1ee78723cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.831002 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07437d4a5a940eb88373c1ee78723cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.837859 1160 ts_tablet_manager.cc:1918] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.838006 1160 log.cc:1198] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/18f85944f66543fe8dd634ad0da0728d
I20250109 20:53:08.838323 1160 ts_tablet_manager.cc:1939] T 18f85944f66543fe8dd634ad0da0728d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.839166 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 18f85944f66543fe8dd634ad0da0728d (table default.Table_9cbf7236e62e4c22a28e6ccb34f76495 [id=f3bf80bcfbb24031a4050f6f53569e93]) successfully deleted
I20250109 20:53:08.839455 1160 tablet_replica.cc:331] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.839679 1160 raft_consensus.cc:2238] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.839892 1160 raft_consensus.cc:2267] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.840032 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a07437d4a5a940eb88373c1ee78723cb (DEFAULT_TABLE table=default.Table_d036d7adca22490991c6e6ca34086cbd [id=f0cdf7c1ef7c47b19956c06f5f61d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.840363 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07437d4a5a940eb88373c1ee78723cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.843370 749 tablet_service.cc:1467] Processing CreateTablet for tablet 1f36060cda664d758ac97749c5528fce (DEFAULT_TABLE table=default.Table_11dc31d4cd4b442bb653a6fb17a590ce [id=2d688692da9c4d2db5c5a34386759067]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.844448 892 tablet_service.cc:1467] Processing CreateTablet for tablet 1f36060cda664d758ac97749c5528fce (DEFAULT_TABLE table=default.Table_11dc31d4cd4b442bb653a6fb17a590ce [id=2d688692da9c4d2db5c5a34386759067]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.845011 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f36060cda664d758ac97749c5528fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.846433 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2802bbb1c8c54c7faa5b7b091bc11a0d (DEFAULT_TABLE table=default.Table_d4029d7ef44145458ee919d3c842bfdb [id=d7d72510f1884533bb8cd8ee72aef71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.846433 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2802bbb1c8c54c7faa5b7b091bc11a0d (DEFAULT_TABLE table=default.Table_d4029d7ef44145458ee919d3c842bfdb [id=d7d72510f1884533bb8cd8ee72aef71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.846750 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2802bbb1c8c54c7faa5b7b091bc11a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.846750 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2802bbb1c8c54c7faa5b7b091bc11a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.847514 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d845bac2539e478083bf7fe6cbc0c717 (table default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b]). Expected version 2 got 0
I20250109 20:53:08.849172 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f36060cda664d758ac97749c5528fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.849195 757 tablet_service.cc:1514] Processing DeleteTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.854342 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 2802bbb1c8c54c7faa5b7b091bc11a0d (DEFAULT_TABLE table=default.Table_d4029d7ef44145458ee919d3c842bfdb [id=d7d72510f1884533bb8cd8ee72aef71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.854663 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2802bbb1c8c54c7faa5b7b091bc11a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.855334 746 tablet_service.cc:1467] Processing CreateTablet for tablet 8e67aa0fafa44a75bceeaff2f05298bc (DEFAULT_TABLE table=default.Table_95fcb9f283734637832d08ce71e8277e [id=b8ea074256ec4c40bfce44bed8541a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.855636 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e67aa0fafa44a75bceeaff2f05298bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.857512 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8e67aa0fafa44a75bceeaff2f05298bc (DEFAULT_TABLE table=default.Table_95fcb9f283734637832d08ce71e8277e [id=b8ea074256ec4c40bfce44bed8541a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.857815 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e67aa0fafa44a75bceeaff2f05298bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.868496 1157 ts_tablet_manager.cc:1918] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.869675 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8e67aa0fafa44a75bceeaff2f05298bc (DEFAULT_TABLE table=default.Table_95fcb9f283734637832d08ce71e8277e [id=b8ea074256ec4c40bfce44bed8541a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.870040 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e67aa0fafa44a75bceeaff2f05298bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.877033 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 1f36060cda664d758ac97749c5528fce (DEFAULT_TABLE table=default.Table_11dc31d4cd4b442bb653a6fb17a590ce [id=2d688692da9c4d2db5c5a34386759067]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.877113 1176 tablet_bootstrap.cc:492] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.877254 1176 ts_tablet_manager.cc:1397] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.172s user 0.004s sys 0.034s
I20250109 20:53:08.877362 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f36060cda664d758ac97749c5528fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.878005 1176 raft_consensus.cc:357] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.878197 1176 raft_consensus.cc:383] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.878249 1176 raft_consensus.cc:738] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.878456 1176 consensus_queue.cc:260] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.879001 1176 ts_tablet_manager.cc:1428] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:08.879297 1176 tablet_bootstrap.cc:492] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.880846 1176 tablet_bootstrap.cc:654] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.883909 1160 ts_tablet_manager.cc:1905] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.892319 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2e36aa38ac6b4bf191f0652d3e3b0691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:08.892505 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_342b51d32c0845ee9786a3b0cd87e309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.893229 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_342b51d32c0845ee9786a3b0cd87e309 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:08.892866 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e36aa38ac6b4bf191f0652d3e3b0691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.898703 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet f2383d9770834b1aae290308b4185299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.902506 1165 tablet_bootstrap.cc:492] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:08.902642 1165 ts_tablet_manager.cc:1397] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.142s user 0.005s sys 0.001s
I20250109 20:53:08.903357 1165 raft_consensus.cc:357] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.903568 1165 raft_consensus.cc:383] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.903615 1165 raft_consensus.cc:738] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.903834 1165 consensus_queue.cc:260] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.904361 1165 ts_tablet_manager.cc:1428] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:08.906208 1165 tablet_bootstrap.cc:492] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:08.874675 2700 raft_consensus.cc:491] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.906383 2700 raft_consensus.cc:513] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.907980 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:08.908138 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:08.908357 1051 raft_consensus.cc:2463] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:08.908691 713 leader_election.cc:304] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:08.909024 2700 leader_election.cc:290] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.909137 2700 raft_consensus.cc:2798] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:08.909193 2700 raft_consensus.cc:491] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:08.909250 2700 raft_consensus.cc:3054] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:08.910233 712 leader_election.cc:343] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.911792 2700 raft_consensus.cc:513] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.912652 2633 raft_consensus.cc:491] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.912777 2633 raft_consensus.cc:513] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.913271 2633 leader_election.cc:290] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.913672 1176 tablet_bootstrap.cc:492] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:08.913713 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:08.913787 1176 ts_tablet_manager.cc:1397] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.007s
I20250109 20:53:08.913867 1051 raft_consensus.cc:3054] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:08.914672 854 leader_election.cc:343] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 9c3af2a00ab04b41915cffc3eb9206bc
I20250109 20:53:08.914686 1176 raft_consensus.cc:357] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.914845 1176 raft_consensus.cc:383] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.914889 1176 raft_consensus.cc:738] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.914865 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:08.915091 1176 consensus_queue.cc:260] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.915532 1176 ts_tablet_manager.cc:1428] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:08.915844 1176 tablet_bootstrap.cc:492] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:08.916193 1051 raft_consensus.cc:2463] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:08.916697 713 leader_election.cc:304] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:08.917021 2692 raft_consensus.cc:2798] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:08.917122 2692 raft_consensus.cc:695] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:08.917336 2692 consensus_queue.cc:237] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.917958 1165 tablet_bootstrap.cc:654] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:08.917969 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 17940f311b0440a59a42cc57f3a0bc4c from {username='slave'} at 127.0.0.1:33768
W20250109 20:53:08.918531 712 leader_election.cc:343] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:08.919407 854 leader_election.cc:343] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 9c3af2a00ab04b41915cffc3eb9206bc
I20250109 20:53:08.919493 854 leader_election.cc:304] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:08.919786 2633 raft_consensus.cc:2743] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.919889 1160 ts_tablet_manager.cc:1918] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:08.919963 1160 log.cc:1198] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/bd3d0bfb39db40758e82be511d033b4c
I20250109 20:53:08.920284 1160 ts_tablet_manager.cc:1939] T bd3d0bfb39db40758e82be511d033b4c P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.920993 895 tablet_service.cc:1514] Processing DeleteTablet for tablet bba303032679462c8b16f706096b6d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.921314 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]) successfully deleted
W20250109 20:53:08.921519 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 111f5e9afc244ad2bd04df715d85bbe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 111f5e9afc244ad2bd04df715d85bbe9 already in progress: creating tablet
W20250109 20:53:08.921705 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 27cb4e6e3a5944bbadb8f1d619d7361c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27cb4e6e3a5944bbadb8f1d619d7361c already in progress: creating tablet
W20250109 20:53:08.921870 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5ced7e77dcb4404d95175dab42615cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ced7e77dcb4404d95175dab42615cee already in progress: creating tablet
W20250109 20:53:08.922029 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b5bcf393703c4c9694190923216585d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bcf393703c4c9694190923216585d9 already in progress: creating tablet
I20250109 20:53:08.922165 1176 tablet_bootstrap.cc:654] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:08.922181 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2d3a253685e34e80a26f338be66d31bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d3a253685e34e80a26f338be66d31bb already in progress: creating tablet
W20250109 20:53:08.922329 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 866f74d8bf534206b6d1642dbb42b603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 866f74d8bf534206b6d1642dbb42b603 already in progress: creating tablet
W20250109 20:53:08.922339 712 consensus_peers.cc:487] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250109 20:53:08.922497 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 59ce2173203d4dc4910065f830b335ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59ce2173203d4dc4910065f830b335ef already in progress: creating tablet
I20250109 20:53:08.934168 895 tablet_service.cc:1467] Processing CreateTablet for tablet 6faba61b888344d5b82f59d0e71ec615 (DEFAULT_TABLE table=default.Table_2383a4fe6f864cf68643fe3d59493438 [id=f9ce2777f2c44d2a8350b85772866381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.934538 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faba61b888344d5b82f59d0e71ec615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.937687 749 tablet_service.cc:1467] Processing CreateTablet for tablet 6faba61b888344d5b82f59d0e71ec615 (DEFAULT_TABLE table=default.Table_2383a4fe6f864cf68643fe3d59493438 [id=f9ce2777f2c44d2a8350b85772866381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.938016 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faba61b888344d5b82f59d0e71ec615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.940023 2700 leader_election.cc:290] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:08.940315 2687 raft_consensus.cc:491] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:08.940433 2687 raft_consensus.cc:513] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:08.941056 2687 leader_election.cc:290] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:08.941912 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f90c364ce844c9ea8eccb4ae23f8dce" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:08.942377 990 leader_election.cc:343] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.944504 1157 log.cc:1198] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6569b450bb7a4a78922061d2ede981fe
I20250109 20:53:08.944908 1157 ts_tablet_manager.cc:1939] T 6569b450bb7a4a78922061d2ede981fe P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:08.944978 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f90c364ce844c9ea8eccb4ae23f8dce" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:08.945454 989 leader_election.cc:343] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:08.945572 989 leader_election.cc:304] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:08.945688 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 6faba61b888344d5b82f59d0e71ec615 (DEFAULT_TABLE table=default.Table_2383a4fe6f864cf68643fe3d59493438 [id=f9ce2777f2c44d2a8350b85772866381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.946075 2687 raft_consensus.cc:2743] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:08.946135 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faba61b888344d5b82f59d0e71ec615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.946869 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1457) took 1179 ms. Trace:
I20250109 20:53:08.947028 1157 rpcz_store.cc:276] 0109 20:53:07.767165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.767225 (+ 60us) service_pool.cc:225] Handling call
0109 20:53:08.724325 (+957100us) ts_tablet_manager.cc:1145] Deleting tablet 6569b450bb7a4a78922061d2ede981fe
0109 20:53:08.724334 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.731460 (+ 7126us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.868415 (+136955us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.946844 (+ 78429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":953313}
I20250109 20:53:08.947607 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6569b450bb7a4a78922061d2ede981fe (table default.Table_416c9e793e9d4fa69bd29e1e30365e06 [id=ee1cb175e24b4968b58f5d87eaeeefb5]) successfully deleted
I20250109 20:53:08.948122 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1459) took 1095 ms. Trace:
I20250109 20:53:08.948279 1157 rpcz_store.cc:276] 0109 20:53:07.852904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.852949 (+ 45us) service_pool.cc:225] Handling call
0109 20:53:08.948013 (+1095064us) ts_tablet_manager.cc:1145] Deleting tablet d914848ad47142f4b64597fd00abdacd
0109 20:53:08.948022 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.948106 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1094881}
I20250109 20:53:08.959013 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_deb4e6bf04ca43d4a1a8dbe6bb09a3da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:08.959532 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_deb4e6bf04ca43d4a1a8dbe6bb09a3da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:08.960623 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_56a50e008101433aba64aa57711f3688" } new_table_name: "default.Table_501c89e3ba4d4bb783a594e619903bcb" modify_external_catalogs: true
I20250109 20:53:08.960865 1166 ts_tablet_manager.cc:1918] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.960976 1166 log.cc:1198] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2db4e85cd4094cf0b5b699402346b980
I20250109 20:53:08.961257 1166 ts_tablet_manager.cc:1939] T 2db4e85cd4094cf0b5b699402346b980 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:53:08.963006 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 23b0e904a87e499f812d208228729baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b0e904a87e499f812d208228729baf already in progress: creating tablet
W20250109 20:53:08.963586 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 4e29d5e170b040328770ef8b3a22d462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e29d5e170b040328770ef8b3a22d462 already in progress: creating tablet
I20250109 20:53:08.964427 1165 tablet_bootstrap.cc:492] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:08.964629 1165 ts_tablet_manager.cc:1397] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.001s
I20250109 20:53:08.965345 1165 raft_consensus.cc:357] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.966023 1165 raft_consensus.cc:383] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.966135 1165 raft_consensus.cc:738] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:08.966444 1165 consensus_queue.cc:260] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.966702 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1417) took 1416 ms. Trace:
I20250109 20:53:08.966812 1166 rpcz_store.cc:276] 0109 20:53:07.550579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.550640 (+ 61us) service_pool.cc:225] Handling call
0109 20:53:08.726316 (+1175676us) ts_tablet_manager.cc:1145] Deleting tablet 2db4e85cd4094cf0b5b699402346b980
0109 20:53:08.726326 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.815315 (+ 88989us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.960744 (+145429us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:08.966678 (+ 5934us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175495}
I20250109 20:53:08.967175 1165 ts_tablet_manager.cc:1428] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:08.967689 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1419) took 1342 ms. Trace:
I20250109 20:53:08.967276 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2db4e85cd4094cf0b5b699402346b980 (table default.Table_e8e4034056dd4856a0bdb4c83d34b6d9 [id=cfe148d699c24f25bcd068a1a0fb0159]) successfully deleted
I20250109 20:53:08.967826 1166 rpcz_store.cc:276] 0109 20:53:07.624867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.624911 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:08.967488 (+1342577us) ts_tablet_manager.cc:1145] Deleting tablet 40546e1b5e854a79a7f706a43b6c6062
0109 20:53:08.967499 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.967669 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1342317}
W20250109 20:53:08.968456 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 40546e1b5e854a79a7f706a43b6c6062 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40546e1b5e854a79a7f706a43b6c6062 already in progress: creating tablet
I20250109 20:53:08.968914 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1418) took 1344 ms. Trace:
W20250109 20:53:08.969465 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d914848ad47142f4b64597fd00abdacd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d914848ad47142f4b64597fd00abdacd already in progress: creating tablet
I20250109 20:53:08.970280 1157 tablet_replica.cc:331] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:08.970803 1157 raft_consensus.cc:2238] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.971040 1157 raft_consensus.cc:2267] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.972055 1157 ts_tablet_manager.cc:1905] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.962903 1158 tablet_bootstrap.cc:492] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:08.973202 1158 ts_tablet_manager.cc:1397] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.179s user 0.000s sys 0.004s
I20250109 20:53:08.973551 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7f2a351186d44db99378b0fc3678356f" } modify_external_catalogs: true
I20250109 20:53:08.973707 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7f2a351186d44db99378b0fc3678356f" } modify_external_catalogs: true
I20250109 20:53:08.974105 1158 raft_consensus.cc:357] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.974289 1158 raft_consensus.cc:383] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:08.974365 1158 raft_consensus.cc:738] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:08.974602 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b895b0c2ba2445583d54b4f515ea7d5 consensus state has no leader
I20250109 20:53:08.974565 1158 consensus_queue.cc:260] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:08.975252 1158 ts_tablet_manager.cc:1428] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:53:08.975561 1158 tablet_bootstrap.cc:492] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:08.978456 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 225046d45c1c43b488babf5da8de75d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.978996 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 07cbc61d839d46d2bae8235d6c92fbbd
I20250109 20:53:08.979903 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 07cbc61d839d46d2bae8235d6c92fbbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.980134 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_93fecf5906e3423684391032f8b408c8" } modify_external_catalogs: true
I20250109 20:53:08.980377 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_93fecf5906e3423684391032f8b408c8" } modify_external_catalogs: true
I20250109 20:53:08.981359 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 07cbc61d839d46d2bae8235d6c92fbbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:08.981628 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9c2eb4c8de43470a9540162b9cf17db7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c2eb4c8de43470a9540162b9cf17db7 already in progress: creating tablet
W20250109 20:53:08.982167 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7c9a4ad54fef41c3941fd9a02be5a418 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c9a4ad54fef41c3941fd9a02be5a418 already in progress: creating tablet
W20250109 20:53:08.982641 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 365d3a2617114ca19de45765a1e95513 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 365d3a2617114ca19de45765a1e95513 already in progress: creating tablet
I20250109 20:53:08.982895 1160 tablet_replica.cc:331] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.983094 1160 raft_consensus.cc:2238] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.983253 1160 raft_consensus.cc:2267] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.987349 1160 ts_tablet_manager.cc:1905] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:08.987653 1165 tablet_bootstrap.cc:492] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:08.988099 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8a22e629234a4456a63b60017767e67c
I20250109 20:53:08.988646 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a22e629234a4456a63b60017767e67c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:08.988960 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a22e629234a4456a63b60017767e67c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:08.990214 746 tablet_service.cc:1467] Processing CreateTablet for tablet d7f78f12e2c348668de816d7446cb9f0 (DEFAULT_TABLE table=default.Table_8801f964bcd24ea5a4c4037abe336954 [id=57d305376b8245be932990ff731bb6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.990489 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e2a32f40c0aa43d8bfb4cc97d297fc76" } modify_external_catalogs: true
I20250109 20:53:08.990527 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8ec506c30a204667919a43f12ab10a09" } modify_external_catalogs: true
I20250109 20:53:08.990697 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e2a32f40c0aa43d8bfb4cc97d297fc76" } modify_external_catalogs: true
I20250109 20:53:08.990767 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8ec506c30a204667919a43f12ab10a09" } modify_external_catalogs: true
I20250109 20:53:08.991885 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c18d342c6c2f416e88d8e3049d4909f1" } modify_external_catalogs: true
I20250109 20:53:08.992024 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c18d342c6c2f416e88d8e3049d4909f1" } modify_external_catalogs: true
I20250109 20:53:08.994119 1160 ts_tablet_manager.cc:1918] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:08.994578 1160 log.cc:1198] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/8539cefebb844b3098a308ca580b0e92
I20250109 20:53:08.994903 1160 ts_tablet_manager.cc:1939] T 8539cefebb844b3098a308ca580b0e92 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:08.995757 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 8539cefebb844b3098a308ca580b0e92 (table default.Table_1d52e3d34dc14e729a515dc24e0787d4 [id=b74c21f2cb874f2485c0e3bc1d6e0dc8]) successfully deleted
W20250109 20:53:08.996327 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a46882e637824d139ac073686c16e46f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a46882e637824d139ac073686c16e46f already in progress: creating tablet
I20250109 20:53:08.996531 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7dacb59e95464e77b88e68386776f17d" } new_table_name: "default.Table_9c8540e2571f47ababd75f8794ea8134" modify_external_catalogs: true
W20250109 20:53:08.996943 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 420f755e81eb4ddd8b872bb7dcea4d13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 420f755e81eb4ddd8b872bb7dcea4d13 already in progress: creating tablet
W20250109 20:53:08.997493 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet edfcbc74f489406297bfa8df228d2787 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edfcbc74f489406297bfa8df228d2787 already in progress: creating tablet
W20250109 20:53:08.997985 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ade6b4ea4ca44f3eb88d3911b0bbb70b already in progress: creating tablet
W20250109 20:53:08.998445 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet dfc973162d464694a4d59423182584ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc973162d464694a4d59423182584ce already in progress: creating tablet
I20250109 20:53:08.998430 895 tablet_service.cc:1467] Processing CreateTablet for tablet d7f78f12e2c348668de816d7446cb9f0 (DEFAULT_TABLE table=default.Table_8801f964bcd24ea5a4c4037abe336954 [id=57d305376b8245be932990ff731bb6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.998693 1160 tablet_replica.cc:331] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:08.998879 1160 raft_consensus.cc:2238] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:08.999037 1160 raft_consensus.cc:2267] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:08.999614 893 tablet_service.cc:1467] Processing CreateTablet for tablet b3d36682639c4cdcb76b4e65dc637129 (DEFAULT_TABLE table=default.Table_5c3d7ff980904543b065367395417f1b [id=7e4d059e863142f8828e3e3782a098b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:08.998737 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f78f12e2c348668de816d7446cb9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.000546 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d36682639c4cdcb76b4e65dc637129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.001466 2711 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:53:09.001647 2711 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:53:09.001881 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet fc749eddce6444ee82a1846a19e5c039
I20250109 20:53:09.000669 749 tablet_service.cc:1467] Processing CreateTablet for tablet b3d36682639c4cdcb76b4e65dc637129 (DEFAULT_TABLE table=default.Table_5c3d7ff980904543b065367395417f1b [id=7e4d059e863142f8828e3e3782a098b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.002840 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1210 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:53:09.001320 750 tablet_service.cc:1467] Processing CreateTablet for tablet df25513f31d449e5aea4ccb97e364529 (DEFAULT_TABLE table=default.Table_4856edec9677474ea7a18bc9dc730986 [id=45b93e091c6b4980b5683c2de77ee10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.002453 757 tablet_service.cc:1514] Processing DeleteTablet for tablet fc749eddce6444ee82a1846a19e5c039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.003885 1165 tablet_bootstrap.cc:654] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.005893 883 tablet_service.cc:1514] Processing DeleteTablet for tablet fc749eddce6444ee82a1846a19e5c039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.007635 1160 ts_tablet_manager.cc:1905] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:09.008487 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3df9ce1c679145179722f6c8d0436e03 consensus state has no leader
I20250109 20:53:09.009299 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c4cff74250cd435e8a52c010c11eff9b
I20250109 20:53:09.011806 893 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac6a19667a94f429cae8fccc73e62a5 (DEFAULT_TABLE table=default.Table_3541107fa6a94c9f9a55103ee6dee809 [id=0647287364ae454db607107cffaf44b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.011967 757 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac6a19667a94f429cae8fccc73e62a5 (DEFAULT_TABLE table=default.Table_3541107fa6a94c9f9a55103ee6dee809 [id=0647287364ae454db607107cffaf44b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.012275 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac6a19667a94f429cae8fccc73e62a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.013273 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet eae1da4227e54d17bc98ff68f5c36d4f
I20250109 20:53:09.013329 892 tablet_service.cc:1467] Processing CreateTablet for tablet df25513f31d449e5aea4ccb97e364529 (DEFAULT_TABLE table=default.Table_4856edec9677474ea7a18bc9dc730986 [id=45b93e091c6b4980b5683c2de77ee10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.013612 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df25513f31d449e5aea4ccb97e364529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.013792 741 tablet_service.cc:1514] Processing DeleteTablet for tablet eae1da4227e54d17bc98ff68f5c36d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.013983 883 tablet_service.cc:1514] Processing DeleteTablet for tablet eae1da4227e54d17bc98ff68f5c36d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.015161 455 catalog_manager.cc:5526] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:09.015524 455 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:09.027410 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 829bec4f66c24180afd5271057a7d0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.028573 1160 ts_tablet_manager.cc:1918] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.028791 1160 log.cc:1198] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/9c3af2a00ab04b41915cffc3eb9206bc
I20250109 20:53:09.029127 1160 ts_tablet_manager.cc:1939] T 9c3af2a00ab04b41915cffc3eb9206bc P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:09.030354 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d36682639c4cdcb76b4e65dc637129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.030817 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 9c3af2a00ab04b41915cffc3eb9206bc (table default.Table_14c64344192b45778141de193a1c433d [id=ffee48c2614543b981c241ea4a3dc24e]) successfully deleted
I20250109 20:53:09.031116 1160 tablet_replica.cc:331] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:09.031270 1160 raft_consensus.cc:2238] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.031378 1160 raft_consensus.cc:2267] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.031976 1160 ts_tablet_manager.cc:1905] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.032832 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_71981989b778449aa3784479963a2e24" } new_table_name: "default.Table_afef45d1b4bd4084a1d6480bf3f8cce4" modify_external_catalogs: true
I20250109 20:53:09.033222 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac6a19667a94f429cae8fccc73e62a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.036486 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_491d93ee3be24649b2aa425756d6e2de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.036569 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_38c4823038d94d6bb4aa5b539be2f3a0" } new_table_name: "default.Table_8a5a15c139c343788318cef6a615bf6d" modify_external_catalogs: true
I20250109 20:53:09.036655 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c9df47cadc104f3f915d6e77b01d9e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.037007 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_491d93ee3be24649b2aa425756d6e2de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:09.037052 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9df47cadc104f3f915d6e77b01d9e1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.038378 1160 ts_tablet_manager.cc:1918] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.038513 1160 log.cc:1198] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/bba303032679462c8b16f706096b6d4d
I20250109 20:53:09.036232 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_73033d6778e24742a0a23ec4d8d9b39e" } new_table_name: "default.Table_f60b293d973d4420b9c2e4218b562a64" modify_external_catalogs: true
I20250109 20:53:09.038568 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fb5f69358fee4665bc556a1e0feba9c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.038960 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb5f69358fee4665bc556a1e0feba9c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.039039 1160 ts_tablet_manager.cc:1939] T bba303032679462c8b16f706096b6d4d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:09.039501 2633 raft_consensus.cc:491] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.039600 2633 raft_consensus.cc:513] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.039980 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet bba303032679462c8b16f706096b6d4d (table default.Table_eb7bd4795ee740f0ba9088dc52e7b383 [id=6a25ad9a50db4cd7b62f73b4e268fdae]) successfully deleted
I20250109 20:53:09.040154 2633 leader_election.cc:290] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:09.040236 1160 tablet_replica.cc:331] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:09.040390 1160 raft_consensus.cc:2238] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.040513 1160 raft_consensus.cc:2267] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.041370 1160 ts_tablet_manager.cc:1905] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.041438 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.042771 895 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cff74250cd435e8a52c010c11eff9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.043787 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df25513f31d449e5aea4ccb97e364529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.046840 1160 ts_tablet_manager.cc:1918] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.046988 1160 log.cc:1198] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/225046d45c1c43b488babf5da8de75d7
I20250109 20:53:09.047279 1160 ts_tablet_manager.cc:1939] T 225046d45c1c43b488babf5da8de75d7 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:09.048135 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 225046d45c1c43b488babf5da8de75d7 (table default.Table_711789d44eb242309002efa5bd06ffb1 [id=a638ba7e8a3b4c309df4fd05ca948518]) successfully deleted
W20250109 20:53:09.048677 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 07cbc61d839d46d2bae8235d6c92fbbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07cbc61d839d46d2bae8235d6c92fbbd already in progress: creating tablet
W20250109 20:53:09.049332 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8a22e629234a4456a63b60017767e67c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a22e629234a4456a63b60017767e67c already in progress: creating tablet
W20250109 20:53:09.049830 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet fc749eddce6444ee82a1846a19e5c039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc749eddce6444ee82a1846a19e5c039 already in progress: creating tablet
W20250109 20:53:09.050329 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet eae1da4227e54d17bc98ff68f5c36d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eae1da4227e54d17bc98ff68f5c36d4f already in progress: creating tablet
I20250109 20:53:09.050537 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7352e2b8530a4ec7b6238c8cbe672cfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.050915 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c4cff74250cd435e8a52c010c11eff9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4cff74250cd435e8a52c010c11eff9b already in progress: creating tablet
W20250109 20:53:09.050992 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7352e2b8530a4ec7b6238c8cbe672cfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.051863 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_910bea2f2ba9433d8ff8f837707b8192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.052192 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_910bea2f2ba9433d8ff8f837707b8192 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.053246 2700 raft_consensus.cc:491] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.053372 2700 raft_consensus.cc:513] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:09.054147 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbe1308ad5224ef687ad1ca7780bddc5 consensus state has no leader
I20250109 20:53:09.054420 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ed0986b3b49e4efbb806df16467e6164"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.054741 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed0986b3b49e4efbb806df16467e6164 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.055226 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829bec4f66c24180afd5271057a7d0e8" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:09.055647 712 leader_election.cc:343] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.057411 2700 leader_election.cc:290] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:09.059551 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f78f12e2c348668de816d7446cb9f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:09.059571 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c7ca67649b64d26b9bf8341e6d59647 consensus state has no leader
W20250109 20:53:09.067601 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0cf3e606c6e403793fe7ff8e0796efa consensus state has no leader
I20250109 20:53:09.070163 1158 tablet_bootstrap.cc:654] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.071389 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d291843c6ba74e52b931a8ce480e81a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.071813 2711 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
W20250109 20:53:09.072041 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d291843c6ba74e52b931a8ce480e81a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.072057 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1210 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
I20250109 20:53:09.072602 2105 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
I20250109 20:53:09.073729 749 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cff74250cd435e8a52c010c11eff9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.078359 895 tablet_service.cc:1467] Processing CreateTablet for tablet b10390c9d436418b8e505eea58b91b98 (DEFAULT_TABLE table=default.Table_d9bd23a3dfe646688547a78281f08279 [id=3cc421e1861f4ef29d0af67e12887dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.078472 749 tablet_service.cc:1467] Processing CreateTablet for tablet b10390c9d436418b8e505eea58b91b98 (DEFAULT_TABLE table=default.Table_d9bd23a3dfe646688547a78281f08279 [id=3cc421e1861f4ef29d0af67e12887dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.078755 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10390c9d436418b8e505eea58b91b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.078819 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10390c9d436418b8e505eea58b91b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:08.969766 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df7405e93aa4718a62933ae56af9f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f77f35085ff43a5b7fb6cce7e7a3409) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.981967 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 07cbc61d839d46d2bae8235d6c92fbbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.084794 1166 rpcz_store.cc:276] 0109 20:53:07.624728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.624920 (+ 192us) service_pool.cc:225] Handling call
0109 20:53:08.968792 (+1343872us) ts_tablet_manager.cc:1145] Deleting tablet a1ed66fe3b46456ea784a3471575c052
0109 20:53:08.968803 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:08.968897 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1343560}
I20250109 20:53:08.989827 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a22e629234a4456a63b60017767e67c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.000865 1021 tablet_service.cc:1467] Processing CreateTablet for tablet d7f78f12e2c348668de816d7446cb9f0 (DEFAULT_TABLE table=default.Table_8801f964bcd24ea5a4c4037abe336954 [id=57d305376b8245be932990ff731bb6e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.085285 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1420) took 1457 ms. Trace:
I20250109 20:53:09.085366 1166 rpcz_store.cc:276] 0109 20:53:07.627505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.627554 (+ 49us) service_pool.cc:225] Handling call
0109 20:53:09.085120 (+1457566us) ts_tablet_manager.cc:1145] Deleting tablet 2098d97e88be463c89ad3ea42e26938c
0109 20:53:09.085131 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.085264 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1457401}
I20250109 20:53:09.085551 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f78f12e2c348668de816d7446cb9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.085595 1166 tablet_replica.cc:331] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.085793 1166 raft_consensus.cc:2238] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.085951 1166 raft_consensus.cc:2267] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.086447 1166 ts_tablet_manager.cc:1905] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.087811 2711 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:53:09.004348 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet fc749eddce6444ee82a1846a19e5c039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.004361 1020 tablet_service.cc:1467] Processing CreateTablet for tablet df25513f31d449e5aea4ccb97e364529 (DEFAULT_TABLE table=default.Table_4856edec9677474ea7a18bc9dc730986 [id=45b93e091c6b4980b5683c2de77ee10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.089198 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df25513f31d449e5aea4ccb97e364529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.091852 1166 ts_tablet_manager.cc:1918] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.091940 1166 log.cc:1198] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/9c88f42c15d94099903178218bfac1e5
I20250109 20:53:09.092175 1166 ts_tablet_manager.cc:1939] T 9c88f42c15d94099903178218bfac1e5 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.004926 1023 tablet_service.cc:1467] Processing CreateTablet for tablet b3d36682639c4cdcb76b4e65dc637129 (DEFAULT_TABLE table=default.Table_5c3d7ff980904543b065367395417f1b [id=7e4d059e863142f8828e3e3782a098b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.093005 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d36682639c4cdcb76b4e65dc637129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.018352 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac6a19667a94f429cae8fccc73e62a5 (DEFAULT_TABLE table=default.Table_3541107fa6a94c9f9a55103ee6dee809 [id=0647287364ae454db607107cffaf44b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.031791 2684 raft_consensus.cc:491] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.093282 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac6a19667a94f429cae8fccc73e62a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.065506 2717 raft_consensus.cc:491] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.093405 2717 raft_consensus.cc:513] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:09.016695 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet eae1da4227e54d17bc98ff68f5c36d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:08.984364 2687 raft_consensus.cc:491] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.093930 2717 leader_election.cc:290] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.093869 2687 raft_consensus.cc:513] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:09.094326 2687 leader_election.cc:290] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.042321 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3681efa8785480ca65a51b4b40448f1" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:09.093160 2684 raft_consensus.cc:513] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:09.097162 2684 leader_election.cc:290] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.097805 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374df8c6d1424697a3b3922f777e9870" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.098153 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e457baa02f4e48f8a627ffdb05267fc8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.098408 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d13168ca13c400a8a2d9608293ff8c3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:09.098968 990 leader_election.cc:343] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:09.099118 990 leader_election.cc:343] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:09.099274 990 leader_election.cc:343] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.066818 2715 raft_consensus.cc:491] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.099560 2715 raft_consensus.cc:513] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:09.100016 2715 leader_election.cc:290] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.100278 854 leader_election.cc:304] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:09.100540 2633 raft_consensus.cc:2743] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.101112 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56d2c0706c84195acceb97ef4985c14" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:09.101560 990 leader_election.cc:343] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.092976 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1422) took 1387 ms. Trace:
I20250109 20:53:09.103319 1166 rpcz_store.cc:276] 0109 20:53:07.705645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.705702 (+ 57us) service_pool.cc:225] Handling call
0109 20:53:09.085555 (+1379853us) ts_tablet_manager.cc:1145] Deleting tablet 9c88f42c15d94099903178218bfac1e5
0109 20:53:09.085564 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.089271 (+ 3707us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.091767 (+ 2496us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.092959 (+ 1192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379675}
I20250109 20:53:09.103729 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1423) took 1357 ms. Trace:
I20250109 20:53:09.103857 1166 rpcz_store.cc:276] 0109 20:53:07.746148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.746372 (+ 224us) service_pool.cc:225] Handling call
0109 20:53:09.103634 (+1357262us) ts_tablet_manager.cc:1145] Deleting tablet 94ed3ab88fd8470bbbc27d801323dc9d
0109 20:53:09.103642 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.103717 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357055}
I20250109 20:53:09.104177 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1424) took 1357 ms. Trace:
I20250109 20:53:09.104280 1166 rpcz_store.cc:276] 0109 20:53:07.746315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.746620 (+ 305us) service_pool.cc:225] Handling call
0109 20:53:09.104095 (+1357475us) ts_tablet_manager.cc:1145] Deleting tablet dcba62cbadf04761822bf7161c52b758
0109 20:53:09.104104 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.104165 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357328}
I20250109 20:53:09.104543 1166 tablet_replica.cc:331] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.104777 1166 raft_consensus.cc:2238] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.104943 1166 raft_consensus.cc:2267] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.105437 1166 ts_tablet_manager.cc:1905] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.107686 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2f4d959b182d46f388cc60f04895b81e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.108263 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f4d959b182d46f388cc60f04895b81e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.111145 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bdac8179ea73495eb30086eddfba3caa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.111675 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdac8179ea73495eb30086eddfba3caa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.112536 1166 ts_tablet_manager.cc:1918] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.112663 1166 log.cc:1198] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/9170b3a6b9bb446b8a474ae4df249409
I20250109 20:53:09.112980 1166 ts_tablet_manager.cc:1939] T 9170b3a6b9bb446b8a474ae4df249409 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.113585 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1425) took 1302 ms. Trace:
I20250109 20:53:09.113692 1166 rpcz_store.cc:276] 0109 20:53:07.810878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.810958 (+ 80us) service_pool.cc:225] Handling call
0109 20:53:09.104503 (+1293545us) ts_tablet_manager.cc:1145] Deleting tablet 9170b3a6b9bb446b8a474ae4df249409
0109 20:53:09.104512 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.109589 (+ 5077us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.112473 (+ 2884us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.113570 (+ 1097us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293368}
I20250109 20:53:09.114038 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1426) took 1260 ms. Trace:
I20250109 20:53:09.114133 1166 rpcz_store.cc:276] 0109 20:53:07.853567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.853610 (+ 43us) service_pool.cc:225] Handling call
0109 20:53:09.113953 (+1260343us) ts_tablet_manager.cc:1145] Deleting tablet d914848ad47142f4b64597fd00abdacd
0109 20:53:09.113961 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.114026 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1260182}
I20250109 20:53:09.114373 1166 tablet_replica.cc:331] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.114573 1166 raft_consensus.cc:2238] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.114770 1166 raft_consensus.cc:2267] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.115289 1166 ts_tablet_manager.cc:1905] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.122949 750 tablet_service.cc:1467] Processing CreateTablet for tablet f1411ebd8adc4018ae0d0525522ad422 (DEFAULT_TABLE table=default.Table_9a6aaac232ee44669f8b733ec7d73b58 [id=b8b24044aa794a66a271d7f470b857a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.123430 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1411ebd8adc4018ae0d0525522ad422. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.124248 1165 tablet_bootstrap.cc:492] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
W20250109 20:53:09.124778 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a1ed66fe3b46456ea784a3471575c052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ed66fe3b46456ea784a3471575c052 already in progress: creating tablet
W20250109 20:53:09.125197 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2098d97e88be463c89ad3ea42e26938c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2098d97e88be463c89ad3ea42e26938c already in progress: creating tablet
I20250109 20:53:09.125787 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 9c88f42c15d94099903178218bfac1e5 (table default.Table_90ff6b4b3aed4dcf8093176305e600e5 [id=963ec7b3b46e48b18b758570e4b4b681]) successfully deleted
W20250109 20:53:09.126117 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 94ed3ab88fd8470bbbc27d801323dc9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94ed3ab88fd8470bbbc27d801323dc9d already in progress: creating tablet
W20250109 20:53:09.126453 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet dcba62cbadf04761822bf7161c52b758 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcba62cbadf04761822bf7161c52b758 already in progress: creating tablet
I20250109 20:53:09.127274 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 9170b3a6b9bb446b8a474ae4df249409 (table default.Table_c213e6faecce454c879d19b0bd0e787a [id=7236f79dc4c34e848233f315441613d5]) successfully deleted
W20250109 20:53:09.127555 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d914848ad47142f4b64597fd00abdacd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d914848ad47142f4b64597fd00abdacd already in progress: creating tablet
I20250109 20:53:09.122331 1166 ts_tablet_manager.cc:1918] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:09.128176 2715 raft_consensus.cc:491] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.130635 1166 log.cc:1198] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/bd3d0bfb39db40758e82be511d033b4c
I20250109 20:53:09.130713 2715 raft_consensus.cc:513] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.130954 1166 ts_tablet_manager.cc:1939] T bd3d0bfb39db40758e82be511d033b4c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.131464 2715 leader_election.cc:290] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:09.131613 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1427) took 1004 ms. Trace:
I20250109 20:53:09.131680 1166 rpcz_store.cc:276] 0109 20:53:08.127506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.127690 (+ 184us) service_pool.cc:225] Handling call
0109 20:53:09.114343 (+986653us) ts_tablet_manager.cc:1145] Deleting tablet bd3d0bfb39db40758e82be511d033b4c
0109 20:53:09.114349 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.118131 (+ 3782us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.122265 (+ 4134us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.131596 (+ 9331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":986446}
I20250109 20:53:09.131955 1166 tablet_replica.cc:331] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.132112 1166 raft_consensus.cc:2238] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.132225 1166 raft_consensus.cc:2267] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.132510 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.132627 1166 ts_tablet_manager.cc:1905] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:09.133201 990 leader_election.cc:343] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.128274 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cff74250cd435e8a52c010c11eff9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.128470 1020 tablet_service.cc:1467] Processing CreateTablet for tablet b10390c9d436418b8e505eea58b91b98 (DEFAULT_TABLE table=default.Table_d9bd23a3dfe646688547a78281f08279 [id=3cc421e1861f4ef29d0af67e12887dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.134076 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10390c9d436418b8e505eea58b91b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.135335 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]) successfully deleted
I20250109 20:53:09.129469 1176 tablet_bootstrap.cc:492] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:09.136879 1176 ts_tablet_manager.cc:1397] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.221s user 0.018s sys 0.010s
I20250109 20:53:09.137465 1176 raft_consensus.cc:357] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.137622 1176 raft_consensus.cc:383] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.137666 1176 raft_consensus.cc:738] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.137833 1176 consensus_queue.cc:260] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.138059 1166 ts_tablet_manager.cc:1918] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.138137 1166 log.cc:1198] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/a1ee36f3be324b14868976298ca26118
I20250109 20:53:09.138345 1176 ts_tablet_manager.cc:1428] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:09.138379 1166 ts_tablet_manager.cc:1939] T a1ee36f3be324b14868976298ca26118 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.138571 1176 tablet_bootstrap.cc:492] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:09.139654 1176 tablet_bootstrap.cc:654] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.140972 1158 tablet_bootstrap.cc:492] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:09.141086 1158 ts_tablet_manager.cc:1397] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.166s user 0.002s sys 0.001s
I20250109 20:53:09.141714 1158 raft_consensus.cc:357] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.141793 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet a1ee36f3be324b14868976298ca26118 (table default.Table_f8eaab37b5024ccd99aa643e72cdda0a [id=ac6293f1f6fe4760ae7f07e3c925081f]) successfully deleted
I20250109 20:53:09.141978 1158 raft_consensus.cc:383] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.142079 1158 raft_consensus.cc:738] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.142295 1158 consensus_queue.cc:260] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.142792 1158 ts_tablet_manager.cc:1428] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:09.144138 1166 tablet_replica.cc:331] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.146143 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 44af81df98ae4b05927dab7b4246de9f (table default.Table_773def38f29b4a5c9c413cbf28623c4d [id=e802914008204d5c9bbe7732f3c28fd5]). Expected version 2 got 0
I20250109 20:53:09.148828 1166 raft_consensus.cc:2238] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.149067 1166 raft_consensus.cc:2267] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.149823 1166 ts_tablet_manager.cc:1905] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:09.150347 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b5bcf393703c4c9694190923216585d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bcf393703c4c9694190923216585d9 already in progress: creating tablet
W20250109 20:53:09.150533 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 27cb4e6e3a5944bbadb8f1d619d7361c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27cb4e6e3a5944bbadb8f1d619d7361c already in progress: creating tablet
W20250109 20:53:09.150681 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2d3a253685e34e80a26f338be66d31bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d3a253685e34e80a26f338be66d31bb already in progress: creating tablet
W20250109 20:53:09.150815 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 866f74d8bf534206b6d1642dbb42b603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 866f74d8bf534206b6d1642dbb42b603 already in progress: creating tablet
W20250109 20:53:09.150977 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 111f5e9afc244ad2bd04df715d85bbe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 111f5e9afc244ad2bd04df715d85bbe9 already in progress: creating tablet
I20250109 20:53:09.152164 1158 tablet_bootstrap.cc:492] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:09.155597 1158 tablet_bootstrap.cc:654] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.159286 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 50afaf6c42a546b2ad7add93095a7556 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:09.165333 1157 ts_tablet_manager.cc:1918] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.165445 1157 log.cc:1198] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/e716270ffcf44039b075023fd0c2d2a3
W20250109 20:53:09.165585 713 consensus_peers.cc:487] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f -> Peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Couldn't send request to peer 5fcc647ed479491b88e5ffacf2927a1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd3d0bfb39db40758e82be511d033b4c. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:09.165716 1157 ts_tablet_manager.cc:1939] T e716270ffcf44039b075023fd0c2d2a3 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:09.166419 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1460) took 1242 ms. Trace:
I20250109 20:53:09.166496 1157 rpcz_store.cc:276] 0109 20:53:07.923910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:07.925324 (+ 1414us) service_pool.cc:225] Handling call
0109 20:53:08.970219 (+1044895us) ts_tablet_manager.cc:1145] Deleting tablet e716270ffcf44039b075023fd0c2d2a3
0109 20:53:08.970241 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.090024 (+119783us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.165224 (+ 75200us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.166401 (+ 1177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1044290}
I20250109 20:53:09.166857 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet e716270ffcf44039b075023fd0c2d2a3 (table default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]) successfully deleted
I20250109 20:53:09.167166 1157 tablet_replica.cc:331] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:09.167330 1157 raft_consensus.cc:2238] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:09.167606 1157 raft_consensus.cc:2267] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:09.168084 713 leader_election.cc:343] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 829bec4f66c24180afd5271057a7d0e8
I20250109 20:53:09.168210 713 leader_election.cc:304] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:09.168617 2700 raft_consensus.cc:2743] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.169471 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 44af81df98ae4b05927dab7b4246de9f (table default.Table_773def38f29b4a5c9c413cbf28623c4d [id=e802914008204d5c9bbe7732f3c28fd5]). Expected version 2 got 0
I20250109 20:53:09.171854 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 50afaf6c42a546b2ad7add93095a7556
I20250109 20:53:09.172025 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.174088 452 catalog_manager.cc:6596] ReplaceTablet: tablet 50afaf6c42a546b2ad7add93095a7556 deleted and replaced by tablet 51376196a86547c99610385deeb9ffef
I20250109 20:53:09.174366 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 50afaf6c42a546b2ad7add93095a7556 with delete_type TABLET_DATA_DELETED (replaced by tablet 51376196a86547c99610385deeb9ffef) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.174600 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 50afaf6c42a546b2ad7add93095a7556 with delete_type TABLET_DATA_DELETED (replaced by tablet 51376196a86547c99610385deeb9ffef) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.175138 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a6e85bbe71224c05bb4b683d0caa2d57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.175364 1021 tablet_service.cc:1467] Processing CreateTablet for tablet f1411ebd8adc4018ae0d0525522ad422 (DEFAULT_TABLE table=default.Table_9a6aaac232ee44669f8b733ec7d73b58 [id=b8b24044aa794a66a271d7f470b857a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.175575 1176 tablet_bootstrap.cc:492] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
W20250109 20:53:09.175585 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6e85bbe71224c05bb4b683d0caa2d57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.175626 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1411ebd8adc4018ae0d0525522ad422. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.175688 1176 ts_tablet_manager.cc:1397] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.001s
I20250109 20:53:09.176406 1176 raft_consensus.cc:357] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.176571 1176 raft_consensus.cc:383] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.176635 1176 raft_consensus.cc:738] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.176854 1176 consensus_queue.cc:260] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.177306 1176 ts_tablet_manager.cc:1428] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:53:09.177547 1176 tablet_bootstrap.cc:492] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:09.178930 1176 tablet_bootstrap.cc:654] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.181041 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_462eb45aaf31437ba051a4fbf3d53d43" } modify_external_catalogs: true
I20250109 20:53:09.181201 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_462eb45aaf31437ba051a4fbf3d53d43" } modify_external_catalogs: true
I20250109 20:53:09.182389 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_452790ac382d43de97d706e1328a1543" } new_table_name: "default.Table_6495c77a2ea14f0f82b4b9cc4a866f7c" modify_external_catalogs: true
I20250109 20:53:09.183513 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet f8eba3dcab764a98857604b63bf9dce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.124966 895 tablet_service.cc:1467] Processing CreateTablet for tablet f1411ebd8adc4018ae0d0525522ad422 (DEFAULT_TABLE table=default.Table_9a6aaac232ee44669f8b733ec7d73b58 [id=b8b24044aa794a66a271d7f470b857a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.140974 912 raft_consensus.cc:3054] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:09.185562 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1411ebd8adc4018ae0d0525522ad422. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.185710 1166 ts_tablet_manager.cc:1918] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.185823 1166 log.cc:1198] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c70ddcc69b2b42fcbe133978dd3ac05b
I20250109 20:53:09.188246 1166 ts_tablet_manager.cc:1939] T c70ddcc69b2b42fcbe133978dd3ac05b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.160074 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e457baa02f4e48f8a627ffdb05267fc8" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.189287 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99
I20250109 20:53:09.190101 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.190421 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ee5424d3946544f49aa8a1f94a0fbbc2" } modify_external_catalogs: true
I20250109 20:53:09.190552 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ee5424d3946544f49aa8a1f94a0fbbc2" } modify_external_catalogs: true
I20250109 20:53:09.190672 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet c70ddcc69b2b42fcbe133978dd3ac05b (table default.Table_efdc83e87d8b455696e8c0aaffb7ed87 [id=102d590adc1f44028550f3508da63869]) successfully deleted
W20250109 20:53:09.191186 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 already in progress: creating tablet
I20250109 20:53:09.160508 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374df8c6d1424697a3b3922f777e9870" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.191342 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.160761 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56d2c0706c84195acceb97ef4985c14" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.160755 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d13168ca13c400a8a2d9608293ff8c3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.161175 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.173470 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 50afaf6c42a546b2ad7add93095a7556 with delete_type TABLET_DATA_DELETED (replaced by tablet 51376196a86547c99610385deeb9ffef) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.191936 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:09.192727 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 50afaf6c42a546b2ad7add93095a7556 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50afaf6c42a546b2ad7add93095a7556 already in progress: creating tablet
W20250109 20:53:09.193498 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5ced7e77dcb4404d95175dab42615cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ced7e77dcb4404d95175dab42615cee already in progress: creating tablet
W20250109 20:53:09.195096 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 59ce2173203d4dc4910065f830b335ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59ce2173203d4dc4910065f830b335ef already in progress: creating tablet
W20250109 20:53:09.195855 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 23b0e904a87e499f812d208228729baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b0e904a87e499f812d208228729baf already in progress: creating tablet
W20250109 20:53:09.196357 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7db9bca84dfb43f3aca6f12009c4007e consensus state has no leader
I20250109 20:53:09.196965 1166 tablet_replica.cc:331] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.197000 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 34ca0a49aef34c40982ff34464727a5e
I20250109 20:53:09.197153 1166 raft_consensus.cc:2238] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.197535 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ca0a49aef34c40982ff34464727a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.197911 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ca0a49aef34c40982ff34464727a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:09.198048 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 34ca0a49aef34c40982ff34464727a5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34ca0a49aef34c40982ff34464727a5e already in progress: creating tablet
I20250109 20:53:09.198299 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ca0a49aef34c40982ff34464727a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.198539 1166 raft_consensus.cc:2267] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:09.198892 989 leader_election.cc:343] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.199746 989 leader_election.cc:304] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:09.200304 989 leader_election.cc:343] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.168152 1157 ts_tablet_manager.cc:1905] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.201047 989 leader_election.cc:304] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
W20250109 20:53:09.201522 989 leader_election.cc:343] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.201962 989 leader_election.cc:304] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.200470 2714 raft_consensus.cc:2743] T e457baa02f4e48f8a627ffdb05267fc8 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.202416 2715 raft_consensus.cc:2743] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.202528 2715 raft_consensus.cc:2743] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:09.202862 989 leader_election.cc:343] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.203217 989 leader_election.cc:304] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.203538 1165 ts_tablet_manager.cc:1397] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.216s user 0.001s sys 0.004s
I20250109 20:53:09.203675 2716 raft_consensus.cc:2743] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:09.204066 989 leader_election.cc:343] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.204217 1165 raft_consensus.cc:357] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.204450 1165 raft_consensus.cc:383] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.204504 1165 raft_consensus.cc:738] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.204609 989 leader_election.cc:304] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.205360 2716 raft_consensus.cc:2743] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.204769 1166 ts_tablet_manager.cc:1905] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.205514 1158 tablet_bootstrap.cc:492] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:09.205705 1158 ts_tablet_manager.cc:1397] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.004s
I20250109 20:53:09.206317 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cea365f9151b49bd829baf4014653bd9" } modify_external_catalogs: true
I20250109 20:53:09.206523 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cea365f9151b49bd829baf4014653bd9" } modify_external_catalogs: true
I20250109 20:53:09.206876 1158 raft_consensus.cc:357] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.207125 1158 raft_consensus.cc:383] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.207216 1158 raft_consensus.cc:738] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.207461 1158 consensus_queue.cc:260] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.207571 895 tablet_service.cc:1467] Processing CreateTablet for tablet 137ebc7ac9da4156a41faa8b621c4e8b (DEFAULT_TABLE table=default.Table_05225976d35e4c7ea4686a8ed10e7c5d [id=f1bced824d8348c0b5364e2c613e3a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.207947 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137ebc7ac9da4156a41faa8b621c4e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.207949 1158 ts_tablet_manager.cc:1428] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:53:09.210006 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9014025f02f448f7bad5dd2f9cd1d8d8" } new_table_name: "default.Table_5a9ae4e99a994816ae76cba035f1a148" modify_external_catalogs: true
I20250109 20:53:09.213709 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 41f0935f17714ecead18b813ccf38f74
I20250109 20:53:09.214527 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f0935f17714ecead18b813ccf38f74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.214821 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f0935f17714ecead18b813ccf38f74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.214902 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f0935f17714ecead18b813ccf38f74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.215549 446 master_service.cc:946] ReplaceTablet: received request to replace tablet e560e5b490774c288157294ad6f23a66 from {username='slave'} at 127.0.0.1:33768
W20250109 20:53:09.215654 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 41f0935f17714ecead18b813ccf38f74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41f0935f17714ecead18b813ccf38f74 already in progress: creating tablet
I20250109 20:53:09.216662 1166 ts_tablet_manager.cc:1918] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.216780 1166 log.cc:1198] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/31de9638669040db9674d3aa7df32103
I20250109 20:53:09.217061 1166 ts_tablet_manager.cc:1939] T 31de9638669040db9674d3aa7df32103 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:53:09.218446 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c913dc6da1a0401596634d361d025164 consensus state has no leader
I20250109 20:53:09.218833 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c38beb2126e94974a129403bd9109727" } modify_external_catalogs: true
I20250109 20:53:09.218982 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c38beb2126e94974a129403bd9109727" } modify_external_catalogs: true
I20250109 20:53:09.220273 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4870a667cd3e410590923c9e2497af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.225855 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e560e5b490774c288157294ad6f23a66
I20250109 20:53:09.226287 446 catalog_manager.cc:6596] ReplaceTablet: tablet e560e5b490774c288157294ad6f23a66 deleted and replaced by tablet e620635f69b84608b44fca9649738c3b
I20250109 20:53:09.226855 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e560e5b490774c288157294ad6f23a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e620635f69b84608b44fca9649738c3b) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.227409 749 tablet_service.cc:1514] Processing DeleteTablet for tablet e560e5b490774c288157294ad6f23a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e620635f69b84608b44fca9649738c3b) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:09.227800 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e560e5b490774c288157294ad6f23a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e560e5b490774c288157294ad6f23a66 already in progress: creating tablet
I20250109 20:53:09.228638 749 tablet_service.cc:1467] Processing CreateTablet for tablet 137ebc7ac9da4156a41faa8b621c4e8b (DEFAULT_TABLE table=default.Table_05225976d35e4c7ea4686a8ed10e7c5d [id=f1bced824d8348c0b5364e2c613e3a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.228964 757 tablet_service.cc:1467] Processing CreateTablet for tablet c0c58aa8ac2c44f9b90271d6bb8364e1 (DEFAULT_TABLE table=default.Table_e82bee8a019c4cf4b23febe15b5c5f4e [id=8936c1aa976f437b92c41fff14fa171b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.229640 892 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1308ad5224ef687ad1ca7780bddc5 (DEFAULT_TABLE table=default.Table_8a5a15c139c343788318cef6a615bf6d [id=a1db2f57b8ba4cff9032116988299ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.230020 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe1308ad5224ef687ad1ca7780bddc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.227880 1176 tablet_bootstrap.cc:492] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:09.230314 1176 ts_tablet_manager.cc:1397] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.002s
I20250109 20:53:09.230319 741 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1308ad5224ef687ad1ca7780bddc5 (DEFAULT_TABLE table=default.Table_8a5a15c139c343788318cef6a615bf6d [id=a1db2f57b8ba4cff9032116988299ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.230679 752 tablet_service.cc:1467] Processing CreateTablet for tablet b9c617740d294ba1aa614e9615085fbb (DEFAULT_TABLE table=default.Table_0d0e1604420a4a2d9130c7414314649a [id=e7b2c41ce0084f4a9787646be6e604f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.231035 740 tablet_service.cc:1467] Processing CreateTablet for tablet c4c197ff5ab24865bbca61976f089d36 (DEFAULT_TABLE table=default.Table_038811046f5440acab1ca56d6337e98e [id=f850dbefcb4649f6b1ea20b3802319fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.231374 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c4c197ff5ab24865bbca61976f089d36 (DEFAULT_TABLE table=default.Table_038811046f5440acab1ca56d6337e98e [id=f850dbefcb4649f6b1ea20b3802319fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.231467 1176 raft_consensus.cc:357] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.231614 1176 raft_consensus.cc:383] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.229279 1032 tablet_service.cc:1467] Processing CreateTablet for tablet c0c58aa8ac2c44f9b90271d6bb8364e1 (DEFAULT_TABLE table=default.Table_e82bee8a019c4cf4b23febe15b5c5f4e [id=8936c1aa976f437b92c41fff14fa171b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.230150 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b9c617740d294ba1aa614e9615085fbb (DEFAULT_TABLE table=default.Table_0d0e1604420a4a2d9130c7414314649a [id=e7b2c41ce0084f4a9787646be6e604f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.229789 1022 tablet_service.cc:1467] Processing CreateTablet for tablet fbe1308ad5224ef687ad1ca7780bddc5 (DEFAULT_TABLE table=default.Table_8a5a15c139c343788318cef6a615bf6d [id=a1db2f57b8ba4cff9032116988299ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.231659 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c197ff5ab24865bbca61976f089d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.232549 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0c58aa8ac2c44f9b90271d6bb8364e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.232770 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe1308ad5224ef687ad1ca7780bddc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.232916 893 tablet_service.cc:1467] Processing CreateTablet for tablet c0c58aa8ac2c44f9b90271d6bb8364e1 (DEFAULT_TABLE table=default.Table_e82bee8a019c4cf4b23febe15b5c5f4e [id=8936c1aa976f437b92c41fff14fa171b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.233191 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0c58aa8ac2c44f9b90271d6bb8364e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.235474 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c617740d294ba1aa614e9615085fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.237833 1165 consensus_queue.cc:260] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.238344 1165 ts_tablet_manager.cc:1428] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.035s user 0.001s sys 0.000s
I20250109 20:53:09.238512 887 tablet_service.cc:1467] Processing CreateTablet for tablet c4c197ff5ab24865bbca61976f089d36 (DEFAULT_TABLE table=default.Table_038811046f5440acab1ca56d6337e98e [id=f850dbefcb4649f6b1ea20b3802319fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.238581 1165 tablet_bootstrap.cc:492] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:09.238766 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c197ff5ab24865bbca61976f089d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.239974 1165 tablet_bootstrap.cc:654] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.238355 883 tablet_service.cc:1467] Processing CreateTablet for tablet b9c617740d294ba1aa614e9615085fbb (DEFAULT_TABLE table=default.Table_0d0e1604420a4a2d9130c7414314649a [id=e7b2c41ce0084f4a9787646be6e604f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.241129 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c617740d294ba1aa614e9615085fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.242478 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137ebc7ac9da4156a41faa8b621c4e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.242908 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe1308ad5224ef687ad1ca7780bddc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.243319 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c617740d294ba1aa614e9615085fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.231657 1176 raft_consensus.cc:738] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.244071 1176 consensus_queue.cc:260] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.244560 1176 ts_tablet_manager.cc:1428] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.014s user 0.000s sys 0.000s
I20250109 20:53:09.244837 1176 tablet_bootstrap.cc:492] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:09.245182 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c197ff5ab24865bbca61976f089d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.246259 1176 tablet_bootstrap.cc:654] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.250232 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2871457499b64529851a4ec12e80838d
I20250109 20:53:09.228696 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 137ebc7ac9da4156a41faa8b621c4e8b (DEFAULT_TABLE table=default.Table_05225976d35e4c7ea4686a8ed10e7c5d [id=f1bced824d8348c0b5364e2c613e3a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.250591 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137ebc7ac9da4156a41faa8b621c4e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.251212 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 2871457499b64529851a4ec12e80838d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.251250 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 2871457499b64529851a4ec12e80838d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.252095 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0c58aa8ac2c44f9b90271d6bb8364e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.256819 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 2871457499b64529851a4ec12e80838d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:09.257972 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2871457499b64529851a4ec12e80838d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2871457499b64529851a4ec12e80838d already in progress: creating tablet
I20250109 20:53:09.260098 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d1a9999752c4415ca70dcb514d0e0003" } new_table_name: "default.Table_acec19ab3d374b64898015620c6a0187" modify_external_catalogs: true
I20250109 20:53:09.261001 746 tablet_service.cc:1467] Processing CreateTablet for tablet 5056ac74d94e424aa2e7a79a8c9edd86 (DEFAULT_TABLE table=default.Table_1f571e8b256c4b74bd690e2d61870d9f [id=96d7b9e7066c41deae00581acfb3879f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.261191 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet e560e5b490774c288157294ad6f23a66 with delete_type TABLET_DATA_DELETED (replaced by tablet e620635f69b84608b44fca9649738c3b) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.261533 887 tablet_service.cc:1467] Processing CreateTablet for tablet 5056ac74d94e424aa2e7a79a8c9edd86 (DEFAULT_TABLE table=default.Table_1f571e8b256c4b74bd690e2d61870d9f [id=96d7b9e7066c41deae00581acfb3879f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.262017 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5056ac74d94e424aa2e7a79a8c9edd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.263813 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4b14acbb79d24c09be2fe93f8f1e2c16" } new_table_name: "default.Table_1d8eba562d90473badcc6e2c67e8b52b" modify_external_catalogs: true
W20250109 20:53:09.270319 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 613fef2a18fd483fb58ca66c4c1b346e consensus state has no leader
I20250109 20:53:09.272996 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5056ac74d94e424aa2e7a79a8c9edd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.273380 1165 tablet_bootstrap.cc:492] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:09.273505 1165 ts_tablet_manager.cc:1397] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
I20250109 20:53:09.273869 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 5056ac74d94e424aa2e7a79a8c9edd86 (DEFAULT_TABLE table=default.Table_1f571e8b256c4b74bd690e2d61870d9f [id=96d7b9e7066c41deae00581acfb3879f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.274248 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5056ac74d94e424aa2e7a79a8c9edd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.265858 1158 tablet_bootstrap.cc:492] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:09.279067 1165 raft_consensus.cc:357] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.279271 1165 raft_consensus.cc:383] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.279321 1165 raft_consensus.cc:738] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.280102 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_aac6ddcccf8342c0913ef40e3acccfe0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.280154 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b11de435be264c7da80ec30a7e13e7b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.280589 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b11de435be264c7da80ec30a7e13e7b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:09.280916 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aac6ddcccf8342c0913ef40e3acccfe0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.280954 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0393ed5be2c1415eac8ff6d3ac1c5229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.281275 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0393ed5be2c1415eac8ff6d3ac1c5229 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.281916 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8dbd9988a2e0401891d2ae569ca31ffc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.282294 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dbd9988a2e0401891d2ae569ca31ffc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.282249 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4a980fd105b247649ed64a59d33bb362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.282450 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2f4d959b182d46f388cc60f04895b81e" } new_table_name: "default.Table_e9a53e076bfe41ba92e19b480edff7a5" modify_external_catalogs: true
W20250109 20:53:09.282579 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a980fd105b247649ed64a59d33bb362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.284795 1165 consensus_queue.cc:260] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.285053 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ed6dd3916ff3440b8f88fb3f3d56f5bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.285301 1165 ts_tablet_manager.cc:1428] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
W20250109 20:53:09.285389 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed6dd3916ff3440b8f88fb3f3d56f5bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.285573 1165 tablet_bootstrap.cc:492] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:09.285697 1157 ts_tablet_manager.cc:1918] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:09.285809 1157 log.cc:1198] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/bd3d0bfb39db40758e82be511d033b4c
I20250109 20:53:09.286152 1157 ts_tablet_manager.cc:1939] T bd3d0bfb39db40758e82be511d033b4c P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:09.286626 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ecf49f821ca248e2a9e2e642aed433b7" } modify_external_catalogs: true
I20250109 20:53:09.286754 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ecf49f821ca248e2a9e2e642aed433b7" } modify_external_catalogs: true
I20250109 20:53:09.286856 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1462) took 1160 ms. Trace:
I20250109 20:53:09.286928 1157 rpcz_store.cc:276] 0109 20:53:08.126907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.127248 (+ 341us) service_pool.cc:225] Handling call
0109 20:53:09.167114 (+1039866us) ts_tablet_manager.cc:1145] Deleting tablet bd3d0bfb39db40758e82be511d033b4c
0109 20:53:09.167122 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.243644 (+ 76522us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.285614 (+ 41970us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.286838 (+ 1224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039696}
I20250109 20:53:09.287191 1165 tablet_bootstrap.cc:654] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.287328 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1476) took 1064 ms. Trace:
I20250109 20:53:09.287400 1157 rpcz_store.cc:276] 0109 20:53:08.222492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.222545 (+ 53us) service_pool.cc:225] Handling call
0109 20:53:09.287213 (+1064668us) ts_tablet_manager.cc:1145] Deleting tablet 111f5e9afc244ad2bd04df715d85bbe9
0109 20:53:09.287222 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.287314 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064497}
I20250109 20:53:09.288349 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet bd3d0bfb39db40758e82be511d033b4c (table default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]) successfully deleted
W20250109 20:53:09.288563 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 111f5e9afc244ad2bd04df715d85bbe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 111f5e9afc244ad2bd04df715d85bbe9 already in progress: creating tablet
I20250109 20:53:09.288767 1166 tablet_replica.cc:331] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.288794 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 31de9638669040db9674d3aa7df32103 (table default.Table_df97156287614f04b00e64a3ecfb64ce [id=7afdb751b30e46948bdae5ca45b684d7]) successfully deleted
W20250109 20:53:09.288877 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac2ff099cd964e0ea946542a4f3cf619 consensus state has no leader
I20250109 20:53:09.288955 1166 raft_consensus.cc:2238] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.289108 1166 raft_consensus.cc:2267] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.289598 1166 ts_tablet_manager.cc:1905] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.299901 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b81a2f52069e4ca693327ab785e96d1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.300336 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b81a2f52069e4ca693327ab785e96d1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.317095 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a0aae7613aaf4248b4b8ada6e60fe8ea (DEFAULT_TABLE table=default.Table_80bd0d8dd57e4f28a7edde0d3e630867 [id=62cc862bad9b4a6381c0f71954ef88ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.317163 892 tablet_service.cc:1467] Processing CreateTablet for tablet a0aae7613aaf4248b4b8ada6e60fe8ea (DEFAULT_TABLE table=default.Table_80bd0d8dd57e4f28a7edde0d3e630867 [id=62cc862bad9b4a6381c0f71954ef88ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.317539 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0aae7613aaf4248b4b8ada6e60fe8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.320457 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0aae7613aaf4248b4b8ada6e60fe8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.320487 1166 ts_tablet_manager.cc:1918] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.321180 1166 log.cc:1198] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/374df8c6d1424697a3b3922f777e9870
I20250109 20:53:09.321463 1166 ts_tablet_manager.cc:1939] T 374df8c6d1424697a3b3922f777e9870 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.322340 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 374df8c6d1424697a3b3922f777e9870 (table default.Table_9ef276a3879c406abf2b9f0a326cde2a [id=f1cdb4acec9a469cafbd3e517556a3ad]) successfully deleted
I20250109 20:53:09.287606 1157 tablet_replica.cc:331] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:09.322846 1157 raft_consensus.cc:2238] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.323004 1157 raft_consensus.cc:2267] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.292870 1158 tablet_bootstrap.cc:654] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.323935 1157 ts_tablet_manager.cc:1905] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.316442 741 tablet_service.cc:1467] Processing CreateTablet for tablet a0aae7613aaf4248b4b8ada6e60fe8ea (DEFAULT_TABLE table=default.Table_80bd0d8dd57e4f28a7edde0d3e630867 [id=62cc862bad9b4a6381c0f71954ef88ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.325865 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0aae7613aaf4248b4b8ada6e60fe8ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:09.330672 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 4e29d5e170b040328770ef8b3a22d462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e29d5e170b040328770ef8b3a22d462 already in progress: creating tablet
W20250109 20:53:09.331808 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9c2eb4c8de43470a9540162b9cf17db7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c2eb4c8de43470a9540162b9cf17db7 already in progress: creating tablet
I20250109 20:53:09.316349 740 tablet_service.cc:1467] Processing CreateTablet for tablet 355a7dc898bf4b4c9bb49e63cf999017 (DEFAULT_TABLE table=default.Table_7eaae83ca44c4bdcae00fed3b9e2a6d8 [id=179065284eee4c61a3c0703081a093d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.332388 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355a7dc898bf4b4c9bb49e63cf999017. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:09.332420 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a2e7e76894d4ca29d8e3f439cfd750e has no consensus state
W20250109 20:53:09.332872 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 365d3a2617114ca19de45765a1e95513 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 365d3a2617114ca19de45765a1e95513 already in progress: creating tablet
W20250109 20:53:09.333803 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7c9a4ad54fef41c3941fd9a02be5a418 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c9a4ad54fef41c3941fd9a02be5a418 already in progress: creating tablet
I20250109 20:53:09.334484 1166 tablet_replica.cc:331] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.335387 1166 raft_consensus.cc:2238] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.335800 1166 raft_consensus.cc:2267] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.336726 1166 ts_tablet_manager.cc:1905] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.337303 2714 raft_consensus.cc:491] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.340574 2714 raft_consensus.cc:513] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.341614 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b4c038573a4afa879ecbcba75c35a7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:09.342141 990 leader_election.cc:343] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.342458 2714 leader_election.cc:290] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:09.340057 2716 raft_consensus.cc:491] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.340317 1176 tablet_bootstrap.cc:492] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:09.343016 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b4c038573a4afa879ecbcba75c35a7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.343106 1176 ts_tablet_manager.cc:1397] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.000s
I20250109 20:53:09.343202 2716 raft_consensus.cc:513] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.343801 1176 raft_consensus.cc:357] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.343955 1176 raft_consensus.cc:383] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.344002 1176 raft_consensus.cc:738] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.344177 1176 consensus_queue.cc:260] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.344399 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beca5f1ed9d3434fb7de90cd74a6f09b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.344594 1176 ts_tablet_manager.cc:1428] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250109 20:53:09.344903 990 leader_election.cc:343] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.345163 2716 leader_election.cc:290] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:53:09.345497 989 leader_election.cc:343] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.345621 989 leader_election.cc:304] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.346168 2716 raft_consensus.cc:2743] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.346231 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beca5f1ed9d3434fb7de90cd74a6f09b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:09.346650 989 leader_election.cc:343] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.346747 989 leader_election.cc:304] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.344918 1176 tablet_bootstrap.cc:492] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:09.347210 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 355a7dc898bf4b4c9bb49e63cf999017 (DEFAULT_TABLE table=default.Table_7eaae83ca44c4bdcae00fed3b9e2a6d8 [id=179065284eee4c61a3c0703081a093d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.347474 2714 raft_consensus.cc:2743] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.348623 1176 tablet_bootstrap.cc:654] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.351060 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8081d188eff1448ea8173067e50f6384
I20250109 20:53:09.351824 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ee437300414eccba9926e824f72f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.355537 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu pid 1922
I20250109 20:53:09.355901 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ea073eb0f15f4f89a384b8ff2ce8224f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.356374 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea073eb0f15f4f89a384b8ff2ce8224f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.357967 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_86af9b6709a0449fb4569b630a683ea1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.358454 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86af9b6709a0449fb4569b630a683ea1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.364449 887 tablet_service.cc:1467] Processing CreateTablet for tablet 355a7dc898bf4b4c9bb49e63cf999017 (DEFAULT_TABLE table=default.Table_7eaae83ca44c4bdcae00fed3b9e2a6d8 [id=179065284eee4c61a3c0703081a093d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.364920 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355a7dc898bf4b4c9bb49e63cf999017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.368346 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355a7dc898bf4b4c9bb49e63cf999017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.373345 892 tablet_service.cc:1467] Processing CreateTablet for tablet 146725c0660e4af9abf0256c66dd89e2 (DEFAULT_TABLE table=default.Table_f9795a157f5045f4bd26f3db299b99ac [id=bfc9438255994bd9a5589d4988afb78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.373909 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146725c0660e4af9abf0256c66dd89e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.374192 893 tablet_service.cc:1467] Processing CreateTablet for tablet ff8f8682d11f47ec8300cd1bb7031f20 (DEFAULT_TABLE table=default.Table_f4960d18c1184100bed547ae248a1b47 [id=1a9437273f5546e7b28d7c116a0ba643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.374193 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 146725c0660e4af9abf0256c66dd89e2 (DEFAULT_TABLE table=default.Table_f9795a157f5045f4bd26f3db299b99ac [id=bfc9438255994bd9a5589d4988afb78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.374509 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8f8682d11f47ec8300cd1bb7031f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.374519 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146725c0660e4af9abf0256c66dd89e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:09.378000 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 613fef2a18fd483fb58ca66c4c1b346e (table default.Table_acec19ab3d374b64898015620c6a0187 [id=b160b9bc7e574e1896000b46a6114ca6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 613fef2a18fd483fb58ca66c4c1b346e consensus state has no leader
I20250109 20:53:09.378594 1166 ts_tablet_manager.cc:1918] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.378691 1166 log.cc:1198] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/12e97ff530df46ae80eb1b5615a55a3f
I20250109 20:53:09.378973 1166 ts_tablet_manager.cc:1939] T 12e97ff530df46ae80eb1b5615a55a3f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.384284 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 12e97ff530df46ae80eb1b5615a55a3f (table default.Table_60c85531239c48349f1931cbcf6b2977 [id=73a9141d6eb942018d73d2c01d601801]) successfully deleted
I20250109 20:53:09.384598 1166 tablet_replica.cc:331] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.384788 1166 raft_consensus.cc:2238] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.384919 1166 raft_consensus.cc:2267] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.385363 1166 ts_tablet_manager.cc:1905] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.391979 893 tablet_service.cc:1467] Processing CreateTablet for tablet 006bafd9086e4932b8246a9cd97ef58c (DEFAULT_TABLE table=default.Table_696d678105f1412788294191037d4aa0 [id=e92df82861e64992821bfa0923088d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.392351 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006bafd9086e4932b8246a9cd97ef58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.397482 1176 tablet_bootstrap.cc:492] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:09.397615 1176 ts_tablet_manager.cc:1397] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20250109 20:53:09.398320 1176 raft_consensus.cc:357] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.398485 1176 raft_consensus.cc:383] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.398528 1176 raft_consensus.cc:738] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.398747 1176 consensus_queue.cc:260] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.401434 1176 ts_tablet_manager.cc:1428] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:09.402619 1158 tablet_bootstrap.cc:492] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:09.402761 1158 ts_tablet_manager.cc:1397] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.137s user 0.001s sys 0.004s
I20250109 20:53:09.409799 1166 ts_tablet_manager.cc:1918] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.409924 1166 log.cc:1198] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/75cdf5b737504d4e8f598cd2b265192e
I20250109 20:53:09.410208 1166 ts_tablet_manager.cc:1939] T 75cdf5b737504d4e8f598cd2b265192e P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.410620 1025 tablet_service.cc:1467] Processing CreateTablet for tablet ff8f8682d11f47ec8300cd1bb7031f20 (DEFAULT_TABLE table=default.Table_f4960d18c1184100bed547ae248a1b47 [id=1a9437273f5546e7b28d7c116a0ba643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.411499 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e340970bd22c4332b51c18771b943166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.411943 451 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:09.412554 741 tablet_service.cc:1467] Processing CreateTablet for tablet 006bafd9086e4932b8246a9cd97ef58c (DEFAULT_TABLE table=default.Table_696d678105f1412788294191037d4aa0 [id=e92df82861e64992821bfa0923088d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.413040 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8402eec50b7e4f5983d31f8b063ddec9" } modify_external_catalogs: true
I20250109 20:53:09.413192 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8402eec50b7e4f5983d31f8b063ddec9" } modify_external_catalogs: true
I20250109 20:53:09.414220 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_da940b25eb974021bef8e7d3aa57dfc0" } new_table_name: "default.Table_a158bfa0f3e14ba780d4af70cc509e40" modify_external_catalogs: true
I20250109 20:53:09.415488 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006bafd9086e4932b8246a9cd97ef58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.415592 452 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 7c4ffd6adbaf4f7083e7ba1a518ed813 (table default.Table_6ad624c51a204e2bbb07ab9b0897bd0c [id=64752eaca1894fe2a1fe46af9a52337a]). Expected version 1 got 0
I20250109 20:53:09.416630 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7352e2b8530a4ec7b6238c8cbe672cfc" } modify_external_catalogs: true
I20250109 20:53:09.416785 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7352e2b8530a4ec7b6238c8cbe672cfc" } modify_external_catalogs: true
W20250109 20:53:09.411983 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e340970bd22c4332b51c18771b943166 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.420117 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:09.420652 452 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 1c65489a602141239c99e8e8dae22ab4 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]). Expected version 4 got 0
I20250109 20:53:09.412627 1157 ts_tablet_manager.cc:1918] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.421521 1157 log.cc:1198] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4814361a1e854451a0ee7469cc6d447b
I20250109 20:53:09.422159 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 006bafd9086e4932b8246a9cd97ef58c (DEFAULT_TABLE table=default.Table_696d678105f1412788294191037d4aa0 [id=e92df82861e64992821bfa0923088d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.422555 1157 ts_tablet_manager.cc:1939] T 4814361a1e854451a0ee7469cc6d447b P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:09.422915 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 59ce2173203d4dc4910065f830b335ef from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:09.423446 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1477) took 1200 ms. Trace:
I20250109 20:53:09.423525 740 tablet_service.cc:1467] Processing CreateTablet for tablet ff8f8682d11f47ec8300cd1bb7031f20 (DEFAULT_TABLE table=default.Table_f4960d18c1184100bed547ae248a1b47 [id=1a9437273f5546e7b28d7c116a0ba643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.424890 883 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1fabfb61c64427b2309c2a3cd5f3c0 (DEFAULT_TABLE table=default.Table_3e19fcbb0db842afafa6cc97810e53ea [id=3c3631ce12ef4bf2a5aa6b17fbadfb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.426327 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8f8682d11f47ec8300cd1bb7031f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.423583 1157 rpcz_store.cc:276] 0109 20:53:08.222624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.222746 (+ 122us) service_pool.cc:225] Handling call
0109 20:53:09.287571 (+1064825us) ts_tablet_manager.cc:1145] Deleting tablet 4814361a1e854451a0ee7469cc6d447b
0109 20:53:09.287579 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.326886 (+ 39307us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.412499 (+ 85613us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.423427 (+ 10928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064687}
I20250109 20:53:09.424640 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1fabfb61c64427b2309c2a3cd5f3c0 (DEFAULT_TABLE table=default.Table_3e19fcbb0db842afafa6cc97810e53ea [id=3c3631ce12ef4bf2a5aa6b17fbadfb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.427592 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c1fabfb61c64427b2309c2a3cd5f3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.429533 1165 tablet_bootstrap.cc:492] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:09.429642 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c1fabfb61c64427b2309c2a3cd5f3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.429688 1165 ts_tablet_manager.cc:1397] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.144s user 0.007s sys 0.013s
I20250109 20:53:09.430380 1165 raft_consensus.cc:357] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.430555 1165 raft_consensus.cc:383] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.430617 1165 raft_consensus.cc:738] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.430805 1165 consensus_queue.cc:260] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.431133 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a499646769274f8aa06cd2d975344794" } modify_external_catalogs: true
I20250109 20:53:09.431354 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a499646769274f8aa06cd2d975344794" } modify_external_catalogs: true
I20250109 20:53:09.431814 1165 ts_tablet_manager.cc:1428] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:09.432057 1165 tablet_bootstrap.cc:492] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:09.432940 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1480) took 1174 ms. Trace:
I20250109 20:53:09.433007 1157 rpcz_store.cc:276] 0109 20:53:08.258215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.258628 (+ 413us) service_pool.cc:225] Handling call
0109 20:53:09.432830 (+1174202us) ts_tablet_manager.cc:1145] Deleting tablet 866f74d8bf534206b6d1642dbb42b603
0109 20:53:09.432837 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.432925 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1173953}
I20250109 20:53:09.433233 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4814361a1e854451a0ee7469cc6d447b (table default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]) successfully deleted
W20250109 20:53:09.433518 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 866f74d8bf534206b6d1642dbb42b603 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 866f74d8bf534206b6d1642dbb42b603 already in progress: creating tablet
I20250109 20:53:09.433537 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1481) took 1175 ms. Trace:
I20250109 20:53:09.433655 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 2aa150310cb147fb90402ffbc124d337
I20250109 20:53:09.433600 1157 rpcz_store.cc:276] 0109 20:53:08.258286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.258915 (+ 629us) service_pool.cc:225] Handling call
0109 20:53:09.433448 (+1174533us) ts_tablet_manager.cc:1145] Deleting tablet 5ced7e77dcb4404d95175dab42615cee
0109 20:53:09.433457 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.433524 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174368}
I20250109 20:53:09.433863 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a95a31468b431fb4304cf6680016da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.434089 1160 tablet_replica.cc:331] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:09.434124 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006bafd9086e4932b8246a9cd97ef58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.434266 1160 raft_consensus.cc:2238] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.437151 1166 tablet_replica.cc:331] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.437323 1166 raft_consensus.cc:2238] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.437463 1166 raft_consensus.cc:2267] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.437718 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 75cdf5b737504d4e8f598cd2b265192e (table default.Table_814844402c5e4defbb723b986171a2a2 [id=a4138fd6ff5246d582fc0d75b890c46f]) successfully deleted
I20250109 20:53:09.437923 1166 ts_tablet_manager.cc:1905] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.439673 1176 tablet_bootstrap.cc:492] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:09.441289 1176 tablet_bootstrap.cc:654] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:09.442473 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa89498a0e554c6b8f69f0c0e407da32 consensus state has no leader
I20250109 20:53:09.434437 1160 raft_consensus.cc:2267] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.443542 1160 ts_tablet_manager.cc:1905] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.443954 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 809e53b469de44b0a7104e8581b61650
I20250109 20:53:09.444478 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_631065e711734e5d9e89c9ee8f3e3d06" } new_table_name: "default.Table_8bda12cd6dcd425b96548fb29a430cd9" modify_external_catalogs: true
I20250109 20:53:09.447404 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 7c4ffd6adbaf4f7083e7ba1a518ed813 (table default.Table_6ad624c51a204e2bbb07ab9b0897bd0c [id=64752eaca1894fe2a1fe46af9a52337a]). Expected version 1 got 0
I20250109 20:53:09.434433 1165 tablet_bootstrap.cc:654] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:09.449648 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5ced7e77dcb4404d95175dab42615cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ced7e77dcb4404d95175dab42615cee already in progress: creating tablet
I20250109 20:53:09.450023 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1479) took 1191 ms. Trace:
I20250109 20:53:09.450103 1157 rpcz_store.cc:276] 0109 20:53:08.258112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.259307 (+ 1195us) service_pool.cc:225] Handling call
0109 20:53:09.449900 (+1190593us) ts_tablet_manager.cc:1145] Deleting tablet 2d3a253685e34e80a26f338be66d31bb
0109 20:53:09.449912 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.450005 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190330}
W20250109 20:53:09.450448 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2d3a253685e34e80a26f338be66d31bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d3a253685e34e80a26f338be66d31bb already in progress: creating tablet
I20250109 20:53:09.450736 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1478) took 1192 ms. Trace:
I20250109 20:53:09.450800 1157 rpcz_store.cc:276] 0109 20:53:08.258023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.259576 (+ 1553us) service_pool.cc:225] Handling call
0109 20:53:09.450639 (+1191063us) ts_tablet_manager.cc:1145] Deleting tablet 27cb4e6e3a5944bbadb8f1d619d7361c
0109 20:53:09.450650 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.450720 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190903}
W20250109 20:53:09.451105 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 27cb4e6e3a5944bbadb8f1d619d7361c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27cb4e6e3a5944bbadb8f1d619d7361c already in progress: creating tablet
I20250109 20:53:09.451385 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1482) took 1193 ms. Trace:
I20250109 20:53:09.451450 1157 rpcz_store.cc:276] 0109 20:53:08.258356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.259751 (+ 1395us) service_pool.cc:225] Handling call
0109 20:53:09.451298 (+1191547us) ts_tablet_manager.cc:1145] Deleting tablet b5bcf393703c4c9694190923216585d9
0109 20:53:09.451306 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.451371 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191395}
W20250109 20:53:09.451723 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b5bcf393703c4c9694190923216585d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bcf393703c4c9694190923216585d9 already in progress: creating tablet
I20250109 20:53:09.451974 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1488) took 1141 ms. Trace:
I20250109 20:53:09.452030 1157 rpcz_store.cc:276] 0109 20:53:08.310227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.310284 (+ 57us) service_pool.cc:225] Handling call
0109 20:53:09.451897 (+1141613us) ts_tablet_manager.cc:1145] Deleting tablet 59ce2173203d4dc4910065f830b335ef
0109 20:53:09.451905 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.451962 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141456}
I20250109 20:53:09.452224 1160 ts_tablet_manager.cc:1918] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.452319 1160 log.cc:1198] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/85a95a31468b431fb4304cf6680016da
W20250109 20:53:09.452332 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 59ce2173203d4dc4910065f830b335ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59ce2173203d4dc4910065f830b335ef already in progress: creating tablet
I20250109 20:53:09.453147 1160 ts_tablet_manager.cc:1939] T 85a95a31468b431fb4304cf6680016da P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:09.454092 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1eee2fc76a024644bdd78f7fd1ef0e62" } modify_external_catalogs: true
I20250109 20:53:09.454222 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1eee2fc76a024644bdd78f7fd1ef0e62" } modify_external_catalogs: true
I20250109 20:53:09.454912 749 tablet_service.cc:1467] Processing CreateTablet for tablet 35fc6c2816a342859e07a52244978186 (DEFAULT_TABLE table=default.Table_880e4245498f4e4d9f134497dbe4afd9 [id=b0dd039421a04c0c90c689a6b1b08f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.455258 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35fc6c2816a342859e07a52244978186. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.456117 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 85a95a31468b431fb4304cf6680016da (table default.Table_3da7386e032a4516b7fa5ae4b2418db3 [id=0485fd054fb14071b7cf6d0fe1b1ee2c]) successfully deleted
I20250109 20:53:09.456677 883 tablet_service.cc:1467] Processing CreateTablet for tablet 35fc6c2816a342859e07a52244978186 (DEFAULT_TABLE table=default.Table_880e4245498f4e4d9f134497dbe4afd9 [id=b0dd039421a04c0c90c689a6b1b08f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.457085 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35fc6c2816a342859e07a52244978186. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.459833 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f03d4e679a1c4793bd83cd979acced88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.460093 1160 tablet_replica.cc:331] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:09.460322 1160 raft_consensus.cc:2238] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.460510 1160 raft_consensus.cc:2267] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.461138 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 04b4247f52ce4d33bf49d9625a1592bf
I20250109 20:53:09.461632 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b4247f52ce4d33bf49d9625a1592bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.462174 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b4247f52ce4d33bf49d9625a1592bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:09.462349 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 751be9de034b47b497a0d7b35fa10ea7 has no consensus state
I20250109 20:53:09.462420 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 35fc6c2816a342859e07a52244978186 (DEFAULT_TABLE table=default.Table_880e4245498f4e4d9f134497dbe4afd9 [id=b0dd039421a04c0c90c689a6b1b08f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.462517 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff8f8682d11f47ec8300cd1bb7031f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.462704 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b4247f52ce4d33bf49d9625a1592bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.463346 1160 ts_tablet_manager.cc:1905] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.464252 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3911b54a040144baadfc4e0088d58e64
I20250109 20:53:09.465155 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3911b54a040144baadfc4e0088d58e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.465350 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3911b54a040144baadfc4e0088d58e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.465570 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3911b54a040144baadfc4e0088d58e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.465922 1166 ts_tablet_manager.cc:1918] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.466006 1166 log.cc:1198] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/b56d2c0706c84195acceb97ef4985c14
I20250109 20:53:09.466467 746 tablet_service.cc:1467] Processing CreateTablet for tablet 146725c0660e4af9abf0256c66dd89e2 (DEFAULT_TABLE table=default.Table_f9795a157f5045f4bd26f3db299b99ac [id=bfc9438255994bd9a5589d4988afb78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.467931 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146725c0660e4af9abf0256c66dd89e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.468926 1166 ts_tablet_manager.cc:1939] T b56d2c0706c84195acceb97ef4985c14 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.469635 2714 raft_consensus.cc:491] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.469763 2714 raft_consensus.cc:513] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.470335 2714 leader_election.cc:290] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.470954 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44af81df98ae4b05927dab7b4246de9f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.471413 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc3102ec03f4e0690535c3b77f3be5f (DEFAULT_TABLE table=default.Table_27aa42c2c837429bbca3a60650ee51ea [id=0aadd4da67eb4acc9fd78565e5b81a0a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:09.471473 990 leader_election.cc:343] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.471688 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc3102ec03f4e0690535c3b77f3be5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.472316 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44af81df98ae4b05927dab7b4246de9f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:09.472726 989 leader_election.cc:343] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.472837 989 leader_election.cc:304] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.473134 2714 raft_consensus.cc:2743] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.475580 2692 raft_consensus.cc:491] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.475668 2692 raft_consensus.cc:513] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.476181 2692 leader_election.cc:290] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:09.476636 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8ea5ea7fa44866b6d7160466f5a896" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:09.477151 712 leader_election.cc:343] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.477706 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc3102ec03f4e0690535c3b77f3be5f (DEFAULT_TABLE table=default.Table_27aa42c2c837429bbca3a60650ee51ea [id=0aadd4da67eb4acc9fd78565e5b81a0a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:09.478008 713 leader_election.cc:343] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: de8ea5ea7fa44866b6d7160466f5a896
I20250109 20:53:09.478114 713 leader_election.cc:304] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:09.478468 2692 raft_consensus.cc:2743] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.479445 750 tablet_service.cc:1467] Processing CreateTablet for tablet dd1ad02446ae4e5f9bab4c58b388267a (DEFAULT_TABLE table=default.Table_1a869e33a3be425aa9015a8491edbbd6 [id=28a012a139134a3b8423fa6232621fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.479699 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd1ad02446ae4e5f9bab4c58b388267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.480877 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b66f03a4dc5241978f3ef4e00ebcd0a0" } new_table_name: "default.Table_4ef8244295a842d5b998b18f6cf71e64" modify_external_catalogs: true
I20250109 20:53:09.481192 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1fabfb61c64427b2309c2a3cd5f3c0 (DEFAULT_TABLE table=default.Table_3e19fcbb0db842afafa6cc97810e53ea [id=3c3631ce12ef4bf2a5aa6b17fbadfb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.481619 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35fc6c2816a342859e07a52244978186. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.482641 1032 tablet_service.cc:1467] Processing CreateTablet for tablet dd1ad02446ae4e5f9bab4c58b388267a (DEFAULT_TABLE table=default.Table_1a869e33a3be425aa9015a8491edbbd6 [id=28a012a139134a3b8423fa6232621fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.482918 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd1ad02446ae4e5f9bab4c58b388267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.483978 893 tablet_service.cc:1467] Processing CreateTablet for tablet dd1ad02446ae4e5f9bab4c58b388267a (DEFAULT_TABLE table=default.Table_1a869e33a3be425aa9015a8491edbbd6 [id=28a012a139134a3b8423fa6232621fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.484302 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd1ad02446ae4e5f9bab4c58b388267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.485671 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet b56d2c0706c84195acceb97ef4985c14 (table default.Table_9849e2509cde44b7ad7db247abe62a79 [id=3064576ec4fb44338fa6ed3200e18ccb]) successfully deleted
I20250109 20:53:09.485942 1166 tablet_replica.cc:331] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.486115 1166 raft_consensus.cc:2238] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.486232 1166 raft_consensus.cc:2267] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.486686 1166 ts_tablet_manager.cc:1905] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.488806 887 tablet_service.cc:1467] Processing CreateTablet for tablet e26e59a0bf3d434393c250ec55de775d (DEFAULT_TABLE table=default.Table_8f43c379140b4354a8c3fe98c89415f1 [id=68a5231164cf4d60964d155d9d5ac843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.488871 757 tablet_service.cc:1467] Processing CreateTablet for tablet e26e59a0bf3d434393c250ec55de775d (DEFAULT_TABLE table=default.Table_8f43c379140b4354a8c3fe98c89415f1 [id=68a5231164cf4d60964d155d9d5ac843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.489204 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e26e59a0bf3d434393c250ec55de775d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.489396 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e26e59a0bf3d434393c250ec55de775d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.494876 1165 tablet_bootstrap.cc:492] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:09.495014 1165 ts_tablet_manager.cc:1397] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.003s
W20250109 20:53:09.495100 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a45cd5ffe7040e385d8cdfb97ed011a consensus state has no leader
I20250109 20:53:09.495663 1165 raft_consensus.cc:357] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.495886 1165 raft_consensus.cc:383] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.495982 1165 raft_consensus.cc:738] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.496212 1165 consensus_queue.cc:260] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.497884 2692 consensus_queue.cc:1035] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:09.499446 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7f412afa12c54777ae20a1441128d6de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.499612 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1489) took 1160 ms. Trace:
I20250109 20:53:09.499671 1157 rpcz_store.cc:276] 0109 20:53:08.339214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.339276 (+ 62us) service_pool.cc:225] Handling call
0109 20:53:09.499504 (+1160228us) ts_tablet_manager.cc:1145] Deleting tablet 23b0e904a87e499f812d208228729baf
0109 20:53:09.499513 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.499596 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160003}
W20250109 20:53:09.499938 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f412afa12c54777ae20a1441128d6de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:09.500000 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 23b0e904a87e499f812d208228729baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b0e904a87e499f812d208228729baf already in progress: creating tablet
I20250109 20:53:09.500301 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1491) took 1063 ms. Trace:
I20250109 20:53:09.500371 1157 rpcz_store.cc:276] 0109 20:53:08.437307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.437355 (+ 48us) service_pool.cc:225] Handling call
0109 20:53:09.500213 (+1062858us) ts_tablet_manager.cc:1145] Deleting tablet 4e29d5e170b040328770ef8b3a22d462
0109 20:53:09.500221 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.500287 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062691}
I20250109 20:53:09.500526 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_333a08dce1d74059a4a06ec1e7eb7513" } new_table_name: "default.Table_942b6d1fc23c4b8294f351a92b81d932" modify_external_catalogs: true
W20250109 20:53:09.500735 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 4e29d5e170b040328770ef8b3a22d462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e29d5e170b040328770ef8b3a22d462 already in progress: creating tablet
I20250109 20:53:09.501044 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1493) took 1021 ms. Trace:
I20250109 20:53:09.501103 1157 rpcz_store.cc:276] 0109 20:53:08.479089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.479137 (+ 48us) service_pool.cc:225] Handling call
0109 20:53:09.500949 (+1021812us) ts_tablet_manager.cc:1145] Deleting tablet 9c2eb4c8de43470a9540162b9cf17db7
0109 20:53:09.500958 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.501030 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998555}
W20250109 20:53:09.501434 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9c2eb4c8de43470a9540162b9cf17db7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c2eb4c8de43470a9540162b9cf17db7 already in progress: creating tablet
I20250109 20:53:09.501727 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1494) took 1018 ms. Trace:
I20250109 20:53:09.501791 1157 rpcz_store.cc:276] 0109 20:53:08.483443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.483498 (+ 55us) service_pool.cc:225] Handling call
0109 20:53:09.501632 (+1018134us) ts_tablet_manager.cc:1145] Deleting tablet 7c9a4ad54fef41c3941fd9a02be5a418
0109 20:53:09.501641 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.501712 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999175}
W20250109 20:53:09.502121 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 7c9a4ad54fef41c3941fd9a02be5a418 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c9a4ad54fef41c3941fd9a02be5a418 already in progress: creating tablet
I20250109 20:53:09.502434 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1492) took 1023 ms. Trace:
I20250109 20:53:09.502506 1157 rpcz_store.cc:276] 0109 20:53:08.478994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.479632 (+ 638us) service_pool.cc:225] Handling call
0109 20:53:09.502336 (+1022704us) ts_tablet_manager.cc:1145] Deleting tablet 365d3a2617114ca19de45765a1e95513
0109 20:53:09.502344 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.502408 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999875}
W20250109 20:53:09.502808 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 365d3a2617114ca19de45765a1e95513 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 365d3a2617114ca19de45765a1e95513 already in progress: creating tablet
I20250109 20:53:09.505241 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 28cd9cd48910497ba3e4b159583cb7a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.505959 1165 ts_tablet_manager.cc:1428] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20250109 20:53:09.509289 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_34fe4c9285164667b3678c3bb7b23c03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.509825 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34fe4c9285164667b3678c3bb7b23c03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.511612 1165 tablet_bootstrap.cc:492] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:09.513329 1165 tablet_bootstrap.cc:654] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.514499 883 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc3102ec03f4e0690535c3b77f3be5f (DEFAULT_TABLE table=default.Table_27aa42c2c837429bbca3a60650ee51ea [id=0aadd4da67eb4acc9fd78565e5b81a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.514917 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc3102ec03f4e0690535c3b77f3be5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.516963 1176 tablet_bootstrap.cc:492] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:09.517129 1176 ts_tablet_manager.cc:1397] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.001s
I20250109 20:53:09.517484 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c1fabfb61c64427b2309c2a3cd5f3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.517704 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc3102ec03f4e0690535c3b77f3be5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.517998 1176 raft_consensus.cc:357] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.518240 1176 raft_consensus.cc:383] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.518311 1176 raft_consensus.cc:738] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.518543 1176 consensus_queue.cc:260] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.519155 1176 ts_tablet_manager.cc:1428] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:09.519430 1176 tablet_bootstrap.cc:492] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:09.521643 749 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1c8c2a0b614f87bc09195c83fac936 (DEFAULT_TABLE table=default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.522086 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d1c8c2a0b614f87bc09195c83fac936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.522125 892 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1c8c2a0b614f87bc09195c83fac936 (DEFAULT_TABLE table=default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.522522 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d1c8c2a0b614f87bc09195c83fac936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.522704 895 tablet_service.cc:1467] Processing CreateTablet for tablet 65ccdbadb4704795857b1cc30d205f31 (DEFAULT_TABLE table=default.Table_9f9eb96e883d4357b65e2bbb4d2da4bf [id=e6f1812a9acc4ae09237d9a06f3a1d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.523041 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ccdbadb4704795857b1cc30d205f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.523188 741 tablet_service.cc:1467] Processing CreateTablet for tablet 65ccdbadb4704795857b1cc30d205f31 (DEFAULT_TABLE table=default.Table_9f9eb96e883d4357b65e2bbb4d2da4bf [id=e6f1812a9acc4ae09237d9a06f3a1d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.523505 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ccdbadb4704795857b1cc30d205f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.523710 755 tablet_service.cc:1467] Processing CreateTablet for tablet ad84125cdb2349b0846e3aab797eacdf (DEFAULT_TABLE table=default.Table_bc2f052f8a5c4572b2f58622e6dc5f29 [id=25ac2a3af14b4675b4e96c615c4f9e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.524132 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad84125cdb2349b0846e3aab797eacdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.528095 743 tablet_service.cc:1467] Processing CreateTablet for tablet ff1817682a754bf3a61a932860a17ecd (DEFAULT_TABLE table=default.Table_5ef3f0f57b0f47d2bd379e2a5710c858 [id=b104410925434c158d4b9168ef481414]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.528508 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1817682a754bf3a61a932860a17ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.529680 1160 ts_tablet_manager.cc:1918] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.530118 1160 log.cc:1198] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/f03d4e679a1c4793bd83cd979acced88
I20250109 20:53:09.530200 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f20593a42961434dbd7ccf432f551dce (DEFAULT_TABLE table=default.Table_9726e60d4f8c433fae0a10b3296d0f8e [id=e6223ecde64e45b8b0e50b1bee92bfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.530423 1160 ts_tablet_manager.cc:1939] T f03d4e679a1c4793bd83cd979acced88 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:09.530542 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20593a42961434dbd7ccf432f551dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.532934 739 tablet_service.cc:1467] Processing CreateTablet for tablet f20593a42961434dbd7ccf432f551dce (DEFAULT_TABLE table=default.Table_9726e60d4f8c433fae0a10b3296d0f8e [id=e6223ecde64e45b8b0e50b1bee92bfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.533015 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 66c8d6632fae4a15889bd7dbb42bb6e1 (DEFAULT_TABLE table=default.Table_b5f34f72cc4e4e16acdc364a7404e3be [id=acd0b8c1a53e433bb542ef9924c4f4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.533211 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20593a42961434dbd7ccf432f551dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.533803 886 tablet_service.cc:1467] Processing CreateTablet for tablet 66c8d6632fae4a15889bd7dbb42bb6e1 (DEFAULT_TABLE table=default.Table_b5f34f72cc4e4e16acdc364a7404e3be [id=acd0b8c1a53e433bb542ef9924c4f4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.533944 884 tablet_service.cc:1467] Processing CreateTablet for tablet e77404f75b22473ba40de0c6854878b7 (DEFAULT_TABLE table=default.Table_026d505e4cfa48c69084663bbc553f83 [id=4e354ce426174bc1bafd4d694a59d20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.534133 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c8d6632fae4a15889bd7dbb42bb6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.534216 892 tablet_service.cc:1467] Processing CreateTablet for tablet f20593a42961434dbd7ccf432f551dce (DEFAULT_TABLE table=default.Table_9726e60d4f8c433fae0a10b3296d0f8e [id=e6223ecde64e45b8b0e50b1bee92bfdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.534355 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77404f75b22473ba40de0c6854878b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.534570 881 tablet_service.cc:1467] Processing CreateTablet for tablet ff1817682a754bf3a61a932860a17ecd (DEFAULT_TABLE table=default.Table_5ef3f0f57b0f47d2bd379e2a5710c858 [id=b104410925434c158d4b9168ef481414]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.534804 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20593a42961434dbd7ccf432f551dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.535019 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1817682a754bf3a61a932860a17ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.536147 1158 raft_consensus.cc:357] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.536314 1158 raft_consensus.cc:383] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.536382 1158 raft_consensus.cc:738] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.536931 745 tablet_service.cc:1467] Processing CreateTablet for tablet 66c8d6632fae4a15889bd7dbb42bb6e1 (DEFAULT_TABLE table=default.Table_b5f34f72cc4e4e16acdc364a7404e3be [id=acd0b8c1a53e433bb542ef9924c4f4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.537209 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c8d6632fae4a15889bd7dbb42bb6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.539899 1158 consensus_queue.cc:260] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.540895 742 tablet_service.cc:1467] Processing CreateTablet for tablet e77404f75b22473ba40de0c6854878b7 (DEFAULT_TABLE table=default.Table_026d505e4cfa48c69084663bbc553f83 [id=4e354ce426174bc1bafd4d694a59d20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.541285 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77404f75b22473ba40de0c6854878b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.541427 1020 tablet_service.cc:1467] Processing CreateTablet for tablet ad84125cdb2349b0846e3aab797eacdf (DEFAULT_TABLE table=default.Table_bc2f052f8a5c4572b2f58622e6dc5f29 [id=25ac2a3af14b4675b4e96c615c4f9e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.541715 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad84125cdb2349b0846e3aab797eacdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.542933 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 65ccdbadb4704795857b1cc30d205f31 (DEFAULT_TABLE table=default.Table_9f9eb96e883d4357b65e2bbb4d2da4bf [id=e6f1812a9acc4ae09237d9a06f3a1d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.543365 895 tablet_service.cc:1467] Processing CreateTablet for tablet ad84125cdb2349b0846e3aab797eacdf (DEFAULT_TABLE table=default.Table_bc2f052f8a5c4572b2f58622e6dc5f29 [id=25ac2a3af14b4675b4e96c615c4f9e2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.543860 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1c8c2a0b614f87bc09195c83fac936 (DEFAULT_TABLE table=default.Table_53a8873bce9e4a749786b79099e7b5b1 [id=b49ad0fe0808440c817b00f7dab76667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.544314 2716 raft_consensus.cc:491] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.544414 2716 raft_consensus.cc:513] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.544945 2716 leader_election.cc:290] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.546177 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8eba3dcab764a98857604b63bf9dce1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.546339 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8eba3dcab764a98857604b63bf9dce1" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:09.546763 990 leader_election.cc:343] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:09.547130 989 leader_election.cc:343] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.547219 989 leader_election.cc:304] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.547495 2716 raft_consensus.cc:2743] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.548066 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9541a8cce37549a2a0c33700443047f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.548760 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3e19fcbb0db842afafa6cc97810e53ea" } new_table_name: "default.Table_7eae384071ce4526b9dc4f5a43b9e184" modify_external_catalogs: true
I20250109 20:53:09.549960 2716 raft_consensus.cc:491] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.550078 2716 raft_consensus.cc:513] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.550570 2716 leader_election.cc:290] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
W20250109 20:53:09.550980 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9541a8cce37549a2a0c33700443047f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.551855 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678ab724db1a42e2b23bbe6cd7652db7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.551925 740 tablet_service.cc:1514] Processing DeleteTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:09.552260 990 leader_election.cc:343] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.552703 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad84125cdb2349b0846e3aab797eacdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.554965 1158 ts_tablet_manager.cc:1428] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250109 20:53:09.558236 2741 tablet.cc:1721] T 5ce0a941f0c14501a947a5c7e2a844e8 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:09.558773 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_dd9b434e7fa540eca7b65552db623183"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.559247 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd9b434e7fa540eca7b65552db623183 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.559746 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d9871f7468704e5982b38ab253e14fef" } modify_external_catalogs: true
I20250109 20:53:09.559875 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d9871f7468704e5982b38ab253e14fef" } modify_external_catalogs: true
I20250109 20:53:09.559832 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4c590bc0e47c49328345d4cee4477b02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.560283 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c590bc0e47c49328345d4cee4477b02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:09.561945 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c1fabfb61c64427b2309c2a3cd5f3c0 consensus state has no leader
I20250109 20:53:09.565032 1157 tablet_replica.cc:331] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:09.565233 1157 raft_consensus.cc:2238] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.565551 1157 raft_consensus.cc:2267] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.566080 1157 ts_tablet_manager.cc:1905] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.567132 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2d0128d7e559487f8176051023f82704"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.567538 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d0128d7e559487f8176051023f82704 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.570176 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet caf806685f914510b53c076f1cfd6bc5
I20250109 20:53:09.571813 752 tablet_service.cc:1514] Processing DeleteTablet for tablet caf806685f914510b53c076f1cfd6bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.572137 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet caf806685f914510b53c076f1cfd6bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.572816 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_da561625a5f54a3c9de94e91dbc12c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.573242 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da561625a5f54a3c9de94e91dbc12c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.574052 1025 tablet_service.cc:1467] Processing CreateTablet for tablet e26e59a0bf3d434393c250ec55de775d (DEFAULT_TABLE table=default.Table_8f43c379140b4354a8c3fe98c89415f1 [id=68a5231164cf4d60964d155d9d5ac843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.578692 1165 tablet_bootstrap.cc:492] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:09.578907 1165 ts_tablet_manager.cc:1397] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20250109 20:53:09.568181 1158 tablet_bootstrap.cc:492] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:09.580302 892 tablet_service.cc:1514] Processing DeleteTablet for tablet caf806685f914510b53c076f1cfd6bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.580696 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet f03d4e679a1c4793bd83cd979acced88 (table default.Table_7592f0da4beb49cda4e5e9c6392a085b [id=5fbc12ffb04546deb78133a5796c63c6]) successfully deleted
W20250109 20:53:09.580952 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 04b4247f52ce4d33bf49d9625a1592bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04b4247f52ce4d33bf49d9625a1592bf already in progress: creating tablet
W20250109 20:53:09.581137 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3911b54a040144baadfc4e0088d58e64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3911b54a040144baadfc4e0088d58e64 already in progress: creating tablet
I20250109 20:53:09.581197 1160 tablet_replica.cc:331] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:09.581395 1160 raft_consensus.cc:2238] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.581579 1160 raft_consensus.cc:2267] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.582265 1160 ts_tablet_manager.cc:1905] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:09.582324 712 consensus_peers.cc:487] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:09.582350 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6] alter complete (version 1)
I20250109 20:53:09.584092 1158 tablet_bootstrap.cc:654] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.583165 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678ab724db1a42e2b23bbe6cd7652db7" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.590654 892 tablet_service.cc:1467] Processing CreateTablet for tablet 4efc3b0c71c748909da82fa0af344ce1 (DEFAULT_TABLE table=default.Table_4f27bb86556f4fc9a4b88a842fa77edb [id=1470b3de17dc48299344724ef9ef83c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.592319 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efc3b0c71c748909da82fa0af344ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.590973 895 tablet_service.cc:1467] Processing CreateTablet for tablet c50325c8a0e34acaa6e2912d3923d43f (DEFAULT_TABLE table=default.Table_662f7ce5426e4f55b64206655147514a [id=26cedb18cc944efcab3597866cd3a977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.594733 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50325c8a0e34acaa6e2912d3923d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.595187 1165 raft_consensus.cc:357] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.596311 1165 raft_consensus.cc:383] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.596876 752 tablet_service.cc:1467] Processing CreateTablet for tablet c50325c8a0e34acaa6e2912d3923d43f (DEFAULT_TABLE table=default.Table_662f7ce5426e4f55b64206655147514a [id=26cedb18cc944efcab3597866cd3a977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.597023 1165 raft_consensus.cc:738] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.597453 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50325c8a0e34acaa6e2912d3923d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.598129 1165 consensus_queue.cc:260] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.602430 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:09.603737 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d1c8c2a0b614f87bc09195c83fac936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.604820 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ccdbadb4704795857b1cc30d205f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.606325 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c8d6632fae4a15889bd7dbb42bb6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.607647 1158 tablet_bootstrap.cc:492] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:09.607760 1158 ts_tablet_manager.cc:1397] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.003s
I20250109 20:53:09.608359 1158 raft_consensus.cc:357] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.608511 1158 raft_consensus.cc:383] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.608556 1158 raft_consensus.cc:738] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.608804 1158 consensus_queue.cc:260] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.609534 1158 ts_tablet_manager.cc:1428] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:09.609994 1158 tablet_bootstrap.cc:492] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:09.609967 887 tablet_service.cc:1467] Processing CreateTablet for tablet fc2937e6375b424b901b7ec413ed9648 (DEFAULT_TABLE table=default.Table_43180160d5a044548b7cf16b6c122bfa [id=c987deabe0ac45748a16183ed6408a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.610255 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2937e6375b424b901b7ec413ed9648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.611255 1158 tablet_bootstrap.cc:654] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:09.591813 989 leader_election.cc:343] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.619834 2695 raft_consensus.cc:491] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.619984 2695 raft_consensus.cc:513] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.620537 2695 leader_election.cc:290] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:09.621176 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce062dc0ac144a19f8fa3eef4a099b4" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:09.621603 712 leader_election.cc:343] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.622045 746 tablet_service.cc:1467] Processing CreateTablet for tablet 4efc3b0c71c748909da82fa0af344ce1 (DEFAULT_TABLE table=default.Table_4f27bb86556f4fc9a4b88a842fa77edb [id=1470b3de17dc48299344724ef9ef83c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.623171 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efc3b0c71c748909da82fa0af344ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.623545 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 4efc3b0c71c748909da82fa0af344ce1 (DEFAULT_TABLE table=default.Table_4f27bb86556f4fc9a4b88a842fa77edb [id=1470b3de17dc48299344724ef9ef83c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.624109 1027 tablet_service.cc:1467] Processing CreateTablet for tablet fc2937e6375b424b901b7ec413ed9648 (DEFAULT_TABLE table=default.Table_43180160d5a044548b7cf16b6c122bfa [id=c987deabe0ac45748a16183ed6408a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.626592 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_515514170bbe43d4ba0791fcc07c725a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.626767 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_23347d9f3f95412a95ad830c547d052e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.627125 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_515514170bbe43d4ba0791fcc07c725a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:09.627183 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23347d9f3f95412a95ad830c547d052e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.627647 1023 tablet_service.cc:1467] Processing CreateTablet for tablet ff1817682a754bf3a61a932860a17ecd (DEFAULT_TABLE table=default.Table_5ef3f0f57b0f47d2bd379e2a5710c858 [id=b104410925434c158d4b9168ef481414]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.627704 989 leader_election.cc:304] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.628441 2716 raft_consensus.cc:491] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.628861 741 tablet_service.cc:1467] Processing CreateTablet for tablet fc2937e6375b424b901b7ec413ed9648 (DEFAULT_TABLE table=default.Table_43180160d5a044548b7cf16b6c122bfa [id=c987deabe0ac45748a16183ed6408a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.629562 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2937e6375b424b901b7ec413ed9648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.636675 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d12ce8f92d214baeb25b460ac371d45e" } modify_external_catalogs: true
I20250109 20:53:09.645665 1165 ts_tablet_manager.cc:1428] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.067s user 0.003s sys 0.000s
I20250109 20:53:09.647791 2741 tablet.cc:1721] T 6619b3dce3ed4325a4871c3d40b42c9d P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:09.654832 1165 tablet_bootstrap.cc:492] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:09.669452 1166 ts_tablet_manager.cc:1918] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.671339 2715 raft_consensus.cc:2743] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.671495 2716 raft_consensus.cc:513] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.672546 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4ffd6adbaf4f7083e7ba1a518ed813" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.672829 1019 tablet_service.cc:1467] Processing CreateTablet for tablet e77404f75b22473ba40de0c6854878b7 (DEFAULT_TABLE table=default.Table_026d505e4cfa48c69084663bbc553f83 [id=4e354ce426174bc1bafd4d694a59d20f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:09.673391 990 leader_election.cc:343] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.673686 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efc3b0c71c748909da82fa0af344ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.674168 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c50325c8a0e34acaa6e2912d3923d43f (DEFAULT_TABLE table=default.Table_662f7ce5426e4f55b64206655147514a [id=26cedb18cc944efcab3597866cd3a977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.674546 1166 log.cc:1198] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/beca5f1ed9d3434fb7de90cd74a6f09b
I20250109 20:53:09.678968 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77404f75b22473ba40de0c6854878b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.678988 2716 leader_election.cc:290] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.679208 1166 ts_tablet_manager.cc:1939] T beca5f1ed9d3434fb7de90cd74a6f09b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.680085 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1465) took 1045 ms. Trace:
I20250109 20:53:09.680183 1166 rpcz_store.cc:276] 0109 20:53:08.634342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.634386 (+ 44us) service_pool.cc:225] Handling call
0109 20:53:09.485903 (+851517us) ts_tablet_manager.cc:1145] Deleting tablet beca5f1ed9d3434fb7de90cd74a6f09b
0109 20:53:09.485911 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.516178 (+ 30267us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.665793 (+149615us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.679864 (+ 14071us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":851348}
I20250109 20:53:09.680507 1166 tablet_replica.cc:331] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.680686 1166 raft_consensus.cc:2238] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.680838 1166 raft_consensus.cc:2267] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.681298 1166 ts_tablet_manager.cc:1905] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.683291 2748 tablet.cc:1721] T 6619b3dce3ed4325a4871c3d40b42c9d P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250109 20:53:09.683779 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 97022.3us
I20250109 20:53:09.684990 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet beca5f1ed9d3434fb7de90cd74a6f09b (table default.Table_c86baf1a40a34eb4beeff2b71b73afde [id=10d4adfc47a7442180591ec9323a64d4]) successfully deleted
I20250109 20:53:09.689081 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1817682a754bf3a61a932860a17ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.692898 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc2937e6375b424b901b7ec413ed9648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.692996 1166 ts_tablet_manager.cc:1918] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.693198 1166 log.cc:1198] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/9d13168ca13c400a8a2d9608293ff8c3
I20250109 20:53:09.693924 1166 ts_tablet_manager.cc:1939] T 9d13168ca13c400a8a2d9608293ff8c3 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.695024 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e26e59a0bf3d434393c250ec55de775d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.698532 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50325c8a0e34acaa6e2912d3923d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.699982 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1466) took 1018 ms. Trace:
I20250109 20:53:09.701360 1166 rpcz_store.cc:276] 0109 20:53:08.681361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.681411 (+ 50us) service_pool.cc:225] Handling call
0109 20:53:09.680448 (+999037us) ts_tablet_manager.cc:1145] Deleting tablet 9d13168ca13c400a8a2d9608293ff8c3
0109 20:53:09.680455 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.683826 (+ 3371us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.692807 (+ 8981us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.699658 (+ 6851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998873}
I20250109 20:53:09.704628 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 9d13168ca13c400a8a2d9608293ff8c3 (table default.Table_ccc647e63d744a0cbfd48bdbd78f5f87 [id=c6c157e5119d4f558b3e2310d2dc3f44]) successfully deleted
W20250109 20:53:09.708540 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a46882e637824d139ac073686c16e46f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a46882e637824d139ac073686c16e46f already in progress: creating tablet
I20250109 20:53:09.710837 2744 tablet.cc:1721] T 5ce0a941f0c14501a947a5c7e2a844e8 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250109 20:53:09.714017 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ade6b4ea4ca44f3eb88d3911b0bbb70b already in progress: creating tablet
I20250109 20:53:09.716624 1160 ts_tablet_manager.cc:1918] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.716768 1160 log.cc:1198] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/28cd9cd48910497ba3e4b159583cb7a6
I20250109 20:53:09.717123 1160 ts_tablet_manager.cc:1939] T 28cd9cd48910497ba3e4b159583cb7a6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:09.717618 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 28cd9cd48910497ba3e4b159583cb7a6 (table default.Table_49bb1f7179c6420c83e530237052e180 [id=38d4e8c8c6294d3983708d35f536e400]) successfully deleted
W20250109 20:53:09.717953 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet caf806685f914510b53c076f1cfd6bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caf806685f914510b53c076f1cfd6bc5 already in progress: creating tablet
I20250109 20:53:09.717655 1158 tablet_bootstrap.cc:492] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:09.720099 446 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 2769e5906c604bd5b14df79db765ec61 (table default.Table_f59d59762b8b4f55b3123c4df5bfd12e [id=3c9451140e48432f8591841d4dfd8751]). Expected version 5 got 0
I20250109 20:53:09.721041 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_116046375f9249cd9761f3719b5967e4" } new_table_name: "default.Table_e8208ae87af74bdc93e6ac8009798d58" modify_external_catalogs: true
I20250109 20:53:09.721292 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:09.722187 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_31cc0ffd3c1c44f3876b4793eae7b026"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.722957 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31cc0ffd3c1c44f3876b4793eae7b026 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.723390 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_85420187c03540a3832414efe59f28d6" } modify_external_catalogs: true
I20250109 20:53:09.723527 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_85420187c03540a3832414efe59f28d6" } modify_external_catalogs: true
I20250109 20:53:09.723956 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d12ce8f92d214baeb25b460ac371d45e" } modify_external_catalogs: true
I20250109 20:53:09.724625 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e61c9814e46a4d32994bdc0e5eaa23ee" } modify_external_catalogs: true
I20250109 20:53:09.724754 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e61c9814e46a4d32994bdc0e5eaa23ee" } modify_external_catalogs: true
I20250109 20:53:09.725890 1165 tablet_bootstrap.cc:654] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.726313 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_020e53b9d9e64e23b63d341a15c2ff78" } new_table_name: "default.Table_047c70a228f5413cb342eeaacc70a66a" modify_external_catalogs: true
I20250109 20:53:09.727524 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_880e4245498f4e4d9f134497dbe4afd9" } modify_external_catalogs: true
I20250109 20:53:09.727629 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_880e4245498f4e4d9f134497dbe4afd9" } modify_external_catalogs: true
I20250109 20:53:09.729825 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4ffd6adbaf4f7083e7ba1a518ed813" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.731431 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bc2073b805334d97bd4e5d2afaf1e0ac" } new_table_name: "default.Table_d449e214145248efa415292c5a4a044b" modify_external_catalogs: true
I20250109 20:53:09.734562 1158 ts_tablet_manager.cc:1397] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.004s
I20250109 20:53:09.737710 1158 raft_consensus.cc:357] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.739500 1158 raft_consensus.cc:383] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.739671 1158 raft_consensus.cc:738] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.739902 1158 consensus_queue.cc:260] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:09.741642 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a9912ba97e04666bba5911cd3d5c296 consensus state has no leader
I20250109 20:53:09.750903 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_588fb629147c47a7abd46370b0537984" } new_table_name: "default.Table_4d5007a07a4749f78933c8230cf483e0" modify_external_catalogs: true
W20250109 20:53:09.752004 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a8e0351fe654fa68e31e4308beb1c79 consensus state has no leader
I20250109 20:53:09.752409 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8de735008fec4af0a7181cc10426842c
I20250109 20:53:09.753249 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 35fc6c2816a342859e07a52244978186
I20250109 20:53:09.754113 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fc6c2816a342859e07a52244978186 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:09.754724 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 35fc6c2816a342859e07a52244978186 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35fc6c2816a342859e07a52244978186 already in progress: creating tablet
W20250109 20:53:09.755184 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1dd16809c9a429e8f0a13aadb929df0 has no consensus state
I20250109 20:53:09.755995 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de735008fec4af0a7181cc10426842c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.747054 2700 raft_consensus.cc:491] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.756227 2700 raft_consensus.cc:513] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.756423 1158 ts_tablet_manager.cc:1428] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
W20250109 20:53:09.756583 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8de735008fec4af0a7181cc10426842c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8de735008fec4af0a7181cc10426842c already in progress: creating tablet
I20250109 20:53:09.756690 1158 tablet_bootstrap.cc:492] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:09.757382 2700 leader_election.cc:290] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:09.757791 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4870a667cd3e410590923c9e2497af54" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.758038 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 678ab724db1a42e2b23bbe6cd7652db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:09.758183 712 leader_election.cc:343] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.759383 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fc6c2816a342859e07a52244978186 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.759382 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de735008fec4af0a7181cc10426842c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.761837 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de735008fec4af0a7181cc10426842c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.762931 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_942b6d1fc23c4b8294f351a92b81d932 [id=c6a5f3d877dc4165b4db160a774f6d3f] alter complete (version 2)
I20250109 20:53:09.764848 1158 tablet_bootstrap.cc:654] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:09.765862 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0e0f4e3a82a437ebbb16426fffa5e2f consensus state has no leader
I20250109 20:53:09.766881 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d1a93305dc21489bb4cc1bb5f36ed408" } new_table_name: "default.Table_cdd109753abb4e82b3fd4b8095308196" modify_external_catalogs: true
I20250109 20:53:09.767783 2749 raft_consensus.cc:491] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.767971 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5d841529b90145caad4c4fd19f9c9669
I20250109 20:53:09.767948 2749 raft_consensus.cc:513] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.768421 2749 leader_election.cc:290] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.769214 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2383d9770834b1aae290308b4185299" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.769289 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d841529b90145caad4c4fd19f9c9669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:09.769726 990 leader_election.cc:343] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:09.769810 713 leader_election.cc:343] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: fce062dc0ac144a19f8fa3eef4a099b4
I20250109 20:53:09.769896 713 leader_election.cc:304] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:09.770190 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2383d9770834b1aae290308b4185299" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.770550 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d841529b90145caad4c4fd19f9c9669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.770598 2695 consensus_queue.cc:1035] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:09.770942 2695 raft_consensus.cc:2743] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.771065 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.771250 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d841529b90145caad4c4fd19f9c9669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:09.771749 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5d841529b90145caad4c4fd19f9c9669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d841529b90145caad4c4fd19f9c9669 already in progress: creating tablet
W20250109 20:53:09.772630 989 leader_election.cc:343] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.772763 989 leader_election.cc:304] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.772971 2749 raft_consensus.cc:2743] T 7c4ffd6adbaf4f7083e7ba1a518ed813 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.773085 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_047c70a228f5413cb342eeaacc70a66a" } modify_external_catalogs: true
I20250109 20:53:09.773375 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_047c70a228f5413cb342eeaacc70a66a" } modify_external_catalogs: true
W20250109 20:53:09.774120 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 068f97bb5a264e539f73cdd415a0e1ab consensus state has no leader
W20250109 20:53:09.776197 989 leader_election.cc:343] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.776366 989 leader_election.cc:304] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.776595 2749 raft_consensus.cc:2743] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:09.781273 713 leader_election.cc:343] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 4870a667cd3e410590923c9e2497af54
I20250109 20:53:09.781410 713 leader_election.cc:304] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:09.781870 2700 raft_consensus.cc:2743] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.785187 1157 ts_tablet_manager.cc:1918] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.785352 1157 log.cc:1198] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/0019a01afbcc452a9a9374c92dc131f3
I20250109 20:53:09.785645 1157 ts_tablet_manager.cc:1939] T 0019a01afbcc452a9a9374c92dc131f3 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:09.786455 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1495) took 1124 ms. Trace:
I20250109 20:53:09.786621 1157 rpcz_store.cc:276] 0109 20:53:08.661893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.661940 (+ 47us) service_pool.cc:225] Handling call
0109 20:53:09.564989 (+903049us) ts_tablet_manager.cc:1145] Deleting tablet 0019a01afbcc452a9a9374c92dc131f3
0109 20:53:09.564997 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.653197 (+ 88200us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.785102 (+131905us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.786433 (+ 1331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":856164}
I20250109 20:53:09.787180 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 0019a01afbcc452a9a9374c92dc131f3 (table default.Table_6d708581ee5c4179af45f5e8ce12bec5 [id=7a4b26558b3b43d097a1c01b0504b037]) successfully deleted
I20250109 20:53:09.787509 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1496) took 1018 ms. Trace:
I20250109 20:53:09.806589 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1468) took 1024 ms. Trace:
I20250109 20:53:09.806764 1166 rpcz_store.cc:276] 0109 20:53:08.781737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.782093 (+ 356us) service_pool.cc:225] Handling call
0109 20:53:09.715653 (+933560us) ts_tablet_manager.cc:1145] Deleting tablet 420f755e81eb4ddd8b872bb7dcea4d13
0109 20:53:09.806390 (+ 90737us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.806568 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":932820}
W20250109 20:53:09.807356 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 420f755e81eb4ddd8b872bb7dcea4d13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 420f755e81eb4ddd8b872bb7dcea4d13 already in progress: creating tablet
I20250109 20:53:09.813037 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f3b977ba7d594fb7aeb874d8b5e7e423"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.813674 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3b977ba7d594fb7aeb874d8b5e7e423 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.808032 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1471) took 1026 ms. Trace:
I20250109 20:53:09.816082 1166 rpcz_store.cc:276] 0109 20:53:08.782044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.782181 (+ 137us) service_pool.cc:225] Handling call
0109 20:53:09.807930 (+1025749us) ts_tablet_manager.cc:1145] Deleting tablet dfc973162d464694a4d59423182584ce
0109 20:53:09.807940 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.808017 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025652}
W20250109 20:53:09.816586 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet dfc973162d464694a4d59423182584ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc973162d464694a4d59423182584ce already in progress: creating tablet
I20250109 20:53:09.817138 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1470) took 1035 ms. Trace:
I20250109 20:53:09.817268 1166 rpcz_store.cc:276] 0109 20:53:08.781942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.782276 (+ 334us) service_pool.cc:225] Handling call
0109 20:53:09.816996 (+1034720us) ts_tablet_manager.cc:1145] Deleting tablet edfcbc74f489406297bfa8df228d2787
0109 20:53:09.817020 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.817120 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1034590}
W20250109 20:53:09.817751 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet edfcbc74f489406297bfa8df228d2787 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edfcbc74f489406297bfa8df228d2787 already in progress: creating tablet
I20250109 20:53:09.818049 1166 tablet_replica.cc:331] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.818251 1166 raft_consensus.cc:2238] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.818411 1166 raft_consensus.cc:2267] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.818894 1166 ts_tablet_manager.cc:1905] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.819325 2744 tablet.cc:1721] T 6619b3dce3ed4325a4871c3d40b42c9d P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:09.828858 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:09.835740 453 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:09.824404 2754 raft_consensus.cc:491] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:53:09.834906 712 consensus_peers.cc:487] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:09.848871 2754 raft_consensus.cc:513] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.849498 2754 leader_election.cc:290] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:09.849740 1157 rpcz_store.cc:276] 0109 20:53:08.769545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.769613 (+ 68us) service_pool.cc:225] Handling call
0109 20:53:09.787416 (+1017803us) ts_tablet_manager.cc:1145] Deleting tablet a46882e637824d139ac073686c16e46f
0109 20:53:09.787424 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.787497 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017641}
I20250109 20:53:09.849915 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea16fcc30f2c4b2eac53f20fcebd043f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:09.850196 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a46882e637824d139ac073686c16e46f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a46882e637824d139ac073686c16e46f already in progress: creating tablet
W20250109 20:53:09.850415 712 leader_election.cc:343] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.850582 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1498) took 1068 ms. Trace:
I20250109 20:53:09.850695 1157 rpcz_store.cc:276] 0109 20:53:08.782504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.782734 (+ 230us) service_pool.cc:225] Handling call
0109 20:53:09.850476 (+1067742us) ts_tablet_manager.cc:1145] Deleting tablet ade6b4ea4ca44f3eb88d3911b0bbb70b
0109 20:53:09.850487 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.850568 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1067571}
W20250109 20:53:09.851073 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ade6b4ea4ca44f3eb88d3911b0bbb70b already in progress: creating tablet
W20250109 20:53:09.851029 713 leader_election.cc:343] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: ea16fcc30f2c4b2eac53f20fcebd043f
I20250109 20:53:09.851191 713 leader_election.cc:304] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:09.851411 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1499) took 1068 ms. Trace:
I20250109 20:53:09.851545 2754 raft_consensus.cc:2743] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.856796 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:09.857975 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fc6c2816a342859e07a52244978186 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.851514 1157 rpcz_store.cc:276] 0109 20:53:08.782599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.782933 (+ 334us) service_pool.cc:225] Handling call
0109 20:53:09.851330 (+1068397us) ts_tablet_manager.cc:1145] Deleting tablet edfcbc74f489406297bfa8df228d2787
0109 20:53:09.851338 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.851398 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068274}
W20250109 20:53:09.866611 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet edfcbc74f489406297bfa8df228d2787 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edfcbc74f489406297bfa8df228d2787 already in progress: creating tablet
I20250109 20:53:09.867194 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1500) took 1084 ms. Trace:
I20250109 20:53:09.870188 2203 generic_service.cc:196] Checking for leaks (request via RPC)
I20250109 20:53:09.873111 1157 rpcz_store.cc:276] 0109 20:53:08.782692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.783082 (+ 390us) service_pool.cc:225] Handling call
0109 20:53:09.867081 (+1083999us) ts_tablet_manager.cc:1145] Deleting tablet dfc973162d464694a4d59423182584ce
0109 20:53:09.867090 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.867180 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083877}
W20250109 20:53:09.873664 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet dfc973162d464694a4d59423182584ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc973162d464694a4d59423182584ce already in progress: creating tablet
I20250109 20:53:09.874120 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1181f2e015334972914ac266d51ac537
I20250109 20:53:09.874341 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1497) took 1091 ms. Trace:
I20250109 20:53:09.874403 1157 rpcz_store.cc:276] 0109 20:53:08.782424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.783222 (+ 798us) service_pool.cc:225] Handling call
0109 20:53:09.874244 (+1091022us) ts_tablet_manager.cc:1145] Deleting tablet 420f755e81eb4ddd8b872bb7dcea4d13
0109 20:53:09.874253 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.874327 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090907}
I20250109 20:53:09.874761 1157 tablet_replica.cc:331] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:09.874950 1157 raft_consensus.cc:2238] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.875159 1157 raft_consensus.cc:2267] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:09.875347 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 420f755e81eb4ddd8b872bb7dcea4d13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 420f755e81eb4ddd8b872bb7dcea4d13 already in progress: creating tablet
I20250109 20:53:09.875950 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1181f2e015334972914ac266d51ac537 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.875972 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1181f2e015334972914ac266d51ac537 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:09.876633 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1181f2e015334972914ac266d51ac537 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1181f2e015334972914ac266d51ac537 already in progress: creating tablet
I20250109 20:53:09.878456 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2a8e0351fe654fa68e31e4308beb1c79
I20250109 20:53:09.879226 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8e0351fe654fa68e31e4308beb1c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:09.879760 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2a8e0351fe654fa68e31e4308beb1c79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a8e0351fe654fa68e31e4308beb1c79 already in progress: creating tablet
I20250109 20:53:09.879956 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8e0351fe654fa68e31e4308beb1c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.881773 1166 ts_tablet_manager.cc:1918] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.881877 1166 log.cc:1198] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/4f90c364ce844c9ea8eccb4ae23f8dce
I20250109 20:53:09.882145 1166 ts_tablet_manager.cc:1939] T 4f90c364ce844c9ea8eccb4ae23f8dce P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:09.882931 1165 tablet_bootstrap.cc:492] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:09.883057 1165 ts_tablet_manager.cc:1397] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.229s user 0.024s sys 0.014s
I20250109 20:53:09.877111 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 1181f2e015334972914ac266d51ac537 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.879684 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8e0351fe654fa68e31e4308beb1c79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.883751 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 012ce4d7dbe14a0795497fd3be2f9feb from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:09.884212 1157 ts_tablet_manager.cc:1905] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.884485 1158 tablet_bootstrap.cc:492] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:09.884584 1158 ts_tablet_manager.cc:1397] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.128s user 0.015s sys 0.019s
I20250109 20:53:09.885978 1165 raft_consensus.cc:357] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.886170 1158 raft_consensus.cc:357] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.886356 1158 raft_consensus.cc:383] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.886420 1158 raft_consensus.cc:738] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.886564 1165 raft_consensus.cc:383] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.886610 1158 consensus_queue.cc:260] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.886830 1165 raft_consensus.cc:738] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.887105 1158 ts_tablet_manager.cc:1428] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:09.887323 1158 tablet_bootstrap.cc:492] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:09.887142 1165 consensus_queue.cc:260] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.888034 1165 ts_tablet_manager.cc:1428] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:53:09.888573 1158 tablet_bootstrap.cc:654] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.888854 1165 tablet_bootstrap.cc:492] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:09.890362 1165 tablet_bootstrap.cc:654] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.896915 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b50c91807734e86840e874886dc0749 with delete_type TABLET_DATA_DELETED (replaced by tablet 39f0cc2be25940d08fad42a8c8d8a07b) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.897750 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 012ce4d7dbe14a0795497fd3be2f9feb
I20250109 20:53:09.898102 451 catalog_manager.cc:6596] ReplaceTablet: tablet 012ce4d7dbe14a0795497fd3be2f9feb deleted and replaced by tablet 0721e92141fe4be482391088bb55c592
I20250109 20:53:09.898429 2749 raft_consensus.cc:491] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.898597 2749 raft_consensus.cc:513] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.898728 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 012ce4d7dbe14a0795497fd3be2f9feb with delete_type TABLET_DATA_DELETED (replaced by tablet 0721e92141fe4be482391088bb55c592) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:09.898766 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 012ce4d7dbe14a0795497fd3be2f9feb with delete_type TABLET_DATA_DELETED (replaced by tablet 0721e92141fe4be482391088bb55c592) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.899168 2749 leader_election.cc:290] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:09.900336 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df7405e93aa4718a62933ae56af9f42" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.900787 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 012ce4d7dbe14a0795497fd3be2f9feb with delete_type TABLET_DATA_DELETED (replaced by tablet 0721e92141fe4be482391088bb55c592) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:09.900871 990 leader_election.cc:343] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.900876 895 tablet_service.cc:1467] Processing CreateTablet for tablet fd0e71f154b642729a5aa33aed1541bc (DEFAULT_TABLE table=default.Table_86af9b6709a0449fb4569b630a683ea1 [id=d2087de8f4de4506bc624e69154b31f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.901160 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df7405e93aa4718a62933ae56af9f42" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:09.901278 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0e71f154b642729a5aa33aed1541bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.903072 1158 tablet_bootstrap.cc:492] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:09.903189 1158 ts_tablet_manager.cc:1397] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250109 20:53:09.903756 1158 raft_consensus.cc:357] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.903926 1158 raft_consensus.cc:383] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:09.903982 1158 raft_consensus.cc:738] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.904165 1158 consensus_queue.cc:260] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.904600 1158 ts_tablet_manager.cc:1428] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250109 20:53:09.905272 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 012ce4d7dbe14a0795497fd3be2f9feb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012ce4d7dbe14a0795497fd3be2f9feb already in progress: creating tablet
W20250109 20:53:09.905635 989 leader_election.cc:343] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:09.905741 989 leader_election.cc:304] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:09.906113 2749 raft_consensus.cc:2743] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.907195 1158 tablet_bootstrap.cc:492] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:09.908591 1158 tablet_bootstrap.cc:654] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:09.908577 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 724d15e05a864852a118badae13a5d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.908797 1160 tablet_replica.cc:331] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:09.908954 1160 raft_consensus.cc:2238] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.909063 1160 raft_consensus.cc:2267] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.909605 1160 ts_tablet_manager.cc:1905] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.911499 741 tablet_service.cc:1467] Processing CreateTablet for tablet fd0e71f154b642729a5aa33aed1541bc (DEFAULT_TABLE table=default.Table_86af9b6709a0449fb4569b630a683ea1 [id=d2087de8f4de4506bc624e69154b31f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.911819 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0e71f154b642729a5aa33aed1541bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.912015 1025 tablet_service.cc:1467] Processing CreateTablet for tablet fd0e71f154b642729a5aa33aed1541bc (DEFAULT_TABLE table=default.Table_86af9b6709a0449fb4569b630a683ea1 [id=d2087de8f4de4506bc624e69154b31f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.912358 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0e71f154b642729a5aa33aed1541bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.913288 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a6cf525cb44c428f7a39a25c4e62dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:09.918743 881 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa0e081c5b455b8ce459b9dfb13bfb (DEFAULT_TABLE table=default.Table_ea073eb0f15f4f89a384b8ff2ce8224f [id=88c5dcaaf6b54627b857b0d66945e0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.919106 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25aa0e081c5b455b8ce459b9dfb13bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.921909 1160 ts_tablet_manager.cc:1918] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.922046 1160 log.cc:1198] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/724d15e05a864852a118badae13a5d3d
I20250109 20:53:09.922338 1160 ts_tablet_manager.cc:1939] T 724d15e05a864852a118badae13a5d3d P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:09.922788 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 724d15e05a864852a118badae13a5d3d (table default.Table_5c9e541186c64e158515aff3e19ccda9 [id=9fcc13c731ac492fbf382b79a3df9ed2]) successfully deleted
I20250109 20:53:09.928517 741 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa0e081c5b455b8ce459b9dfb13bfb (DEFAULT_TABLE table=default.Table_ea073eb0f15f4f89a384b8ff2ce8224f [id=88c5dcaaf6b54627b857b0d66945e0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.928910 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1472) took 1108 ms. Trace:
I20250109 20:53:09.929028 1166 rpcz_store.cc:276] 0109 20:53:08.820499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.820928 (+ 429us) service_pool.cc:225] Handling call
0109 20:53:09.818010 (+997082us) ts_tablet_manager.cc:1145] Deleting tablet 4f90c364ce844c9ea8eccb4ae23f8dce
0109 20:53:09.818018 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.821536 (+ 3518us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.881682 (+ 60146us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.928885 (+ 47203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":996551}
I20250109 20:53:09.929591 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa0e081c5b455b8ce459b9dfb13bfb (DEFAULT_TABLE table=default.Table_ea073eb0f15f4f89a384b8ff2ce8224f [id=88c5dcaaf6b54627b857b0d66945e0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.930408 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25aa0e081c5b455b8ce459b9dfb13bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.936398 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 4f90c364ce844c9ea8eccb4ae23f8dce (table default.Table_fca16c47746e4cbeb149a7d20fd0a979 [id=2a02c7d57a6c4df7ac423f494891b8e8]) successfully deleted
I20250109 20:53:09.936316 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ad137ad7c05d4e4c9a3e84e8f3bb18c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.936480 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f4cefa533b6c443c8aeb122fd2787cc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.936897 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4cefa533b6c443c8aeb122fd2787cc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.936769 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d31a1416d67e49c58ce51ad2c7a4f6c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.936831 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_43c505c1a72e49a5b2d5fa847e9c9402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.937291 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43c505c1a72e49a5b2d5fa847e9c9402 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:09.937317 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d31a1416d67e49c58ce51ad2c7a4f6c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.936780 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7b117f0691724fb8990d5597ce687d65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.937762 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b117f0691724fb8990d5597ce687d65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:09.940291 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad137ad7c05d4e4c9a3e84e8f3bb18c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.940707 1166 tablet_replica.cc:331] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:09.941196 1157 ts_tablet_manager.cc:1918] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.941291 1157 log.cc:1198] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/de8ea5ea7fa44866b6d7160466f5a896
I20250109 20:53:09.941583 1157 ts_tablet_manager.cc:1939] T de8ea5ea7fa44866b6d7160466f5a896 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:09.940438 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_f5728156039c4352b2474ae0d7148de0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.943516 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25aa0e081c5b455b8ce459b9dfb13bfb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:09.943804 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5728156039c4352b2474ae0d7148de0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.944912 2633 raft_consensus.cc:491] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:09.945045 2633 raft_consensus.cc:513] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.946064 2633 leader_election.cc:290] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:09.946177 1166 raft_consensus.cc:2238] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.946328 1166 raft_consensus.cc:2267] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.946851 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce0a941f0c14501a947a5c7e2a844e8" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:09.947257 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce0a941f0c14501a947a5c7e2a844e8" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:09.947858 854 leader_election.cc:304] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:09.948408 2633 raft_consensus.cc:2743] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:09.950139 1166 ts_tablet_manager.cc:1905] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.952214 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5a9ae4e99a994816ae76cba035f1a148" } new_table_name: "default.Table_67241b08848f4a6cb789dca013dffae6" modify_external_catalogs: true
I20250109 20:53:09.953466 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1506) took 1104 ms. Trace:
I20250109 20:53:09.953650 1157 rpcz_store.cc:276] 0109 20:53:08.849106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.849160 (+ 54us) service_pool.cc:225] Handling call
0109 20:53:09.874721 (+1025561us) ts_tablet_manager.cc:1145] Deleting tablet de8ea5ea7fa44866b6d7160466f5a896
0109 20:53:09.874728 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.920062 (+ 45334us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.941105 (+ 21043us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:09.953442 (+ 12337us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025432}
I20250109 20:53:09.954162 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet de8ea5ea7fa44866b6d7160466f5a896 (table default.Table_64ebceb89d2c4e8291bc096091578453 [id=94d89d19277c4e3fbf4ba420ebe0965c]) successfully deleted
W20250109 20:53:09.954732 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 07cbc61d839d46d2bae8235d6c92fbbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07cbc61d839d46d2bae8235d6c92fbbd already in progress: creating tablet
W20250109 20:53:09.954950 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8a22e629234a4456a63b60017767e67c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a22e629234a4456a63b60017767e67c already in progress: creating tablet
W20250109 20:53:09.955144 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet fc749eddce6444ee82a1846a19e5c039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc749eddce6444ee82a1846a19e5c039 already in progress: creating tablet
I20250109 20:53:09.955929 1165 tablet_bootstrap.cc:492] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:09.956053 1165 ts_tablet_manager.cc:1397] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.004s
I20250109 20:53:09.956770 1165 raft_consensus.cc:357] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.956947 1165 raft_consensus.cc:383] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250109 20:53:09.957269 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet eae1da4227e54d17bc98ff68f5c36d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eae1da4227e54d17bc98ff68f5c36d4f already in progress: creating tablet
I20250109 20:53:09.957222 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fab4024b118047b49eddc729dc2f7c63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:09.957546 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f270a526059d4308925d102e914882ed" } new_table_name: "default.Table_07fa5aac1a9d4b4ba13232ba3e4f3571" modify_external_catalogs: true
I20250109 20:53:09.957798 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b53f1d00143b425a941127015bda726a" } modify_external_catalogs: true
I20250109 20:53:09.957944 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b53f1d00143b425a941127015bda726a" } modify_external_catalogs: true
I20250109 20:53:09.958751 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6d59464208774e168bab69b28a04bd5c" } new_table_name: "default.Table_b816c85c0aa34ffeae970f05a3815ebf" modify_external_catalogs: true
I20250109 20:53:09.958889 1165 raft_consensus.cc:738] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:09.959121 1165 consensus_queue.cc:260] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:09.959597 1165 ts_tablet_manager.cc:1428] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:53:09.959924 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e85779bce1cf4ab4926e6c7af5b7fa33" } modify_external_catalogs: true
I20250109 20:53:09.960052 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e85779bce1cf4ab4926e6c7af5b7fa33" } modify_external_catalogs: true
W20250109 20:53:09.957619 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fab4024b118047b49eddc729dc2f7c63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.961284 1165 tablet_bootstrap.cc:492] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:09.963294 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5c3902e3243943718b6fcf1b5d294445"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:09.963678 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c3902e3243943718b6fcf1b5d294445 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:09.965248 1157 tablet_replica.cc:331] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:09.965443 1157 raft_consensus.cc:2238] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.965584 1157 raft_consensus.cc:2267] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.965565 895 tablet_service.cc:1467] Processing CreateTablet for tablet d7e8eec58dce434aaf6aa1b40a4cd18f (DEFAULT_TABLE table=default.Table_b81a2f52069e4ca693327ab785e96d1d [id=e49dc14ae2894fa6844f665ae00178b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.965976 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e8eec58dce434aaf6aa1b40a4cd18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.966142 1157 ts_tablet_manager.cc:1905] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.966537 1021 tablet_service.cc:1467] Processing CreateTablet for tablet cf9cae8f38f94f4aba1fbe8f7cd4b7ed (DEFAULT_TABLE table=default.Table_b11de435be264c7da80ec30a7e13e7b2 [id=224a1537a1e04018b1d58f03927c0452]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.966854 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf9cae8f38f94f4aba1fbe8f7cd4b7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.966962 741 tablet_service.cc:1467] Processing CreateTablet for tablet d7e8eec58dce434aaf6aa1b40a4cd18f (DEFAULT_TABLE table=default.Table_b81a2f52069e4ca693327ab785e96d1d [id=e49dc14ae2894fa6844f665ae00178b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.967274 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e8eec58dce434aaf6aa1b40a4cd18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.969568 749 tablet_service.cc:1467] Processing CreateTablet for tablet cf9cae8f38f94f4aba1fbe8f7cd4b7ed (DEFAULT_TABLE table=default.Table_b11de435be264c7da80ec30a7e13e7b2 [id=224a1537a1e04018b1d58f03927c0452]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.969862 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf9cae8f38f94f4aba1fbe8f7cd4b7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.971624 892 tablet_service.cc:1467] Processing CreateTablet for tablet cf9cae8f38f94f4aba1fbe8f7cd4b7ed (DEFAULT_TABLE table=default.Table_b11de435be264c7da80ec30a7e13e7b2 [id=224a1537a1e04018b1d58f03927c0452]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.972013 887 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3e7bffaac64167aae4974f379b4650 (DEFAULT_TABLE table=default.Table_ed6dd3916ff3440b8f88fb3f3d56f5bf [id=27aceae056004df6a1b7fc88fa5c099d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.972333 883 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2d3dfb96c24a0e84e9a7e19ccf195f (DEFAULT_TABLE table=default.Table_4a980fd105b247649ed64a59d33bb362 [id=1260ee8d2f494ff8a675c73966adaf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.972666 893 tablet_service.cc:1467] Processing CreateTablet for tablet 627a7b307bbd4f1590d5ccd2595a0aaa (DEFAULT_TABLE table=default.Table_8dbd9988a2e0401891d2ae569ca31ffc [id=1a1a1175d0474959a1cf915612e0b31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.973049 886 tablet_service.cc:1467] Processing CreateTablet for tablet d4c014521bc04cbb96e52855febc7439 (DEFAULT_TABLE table=default.Table_bdac8179ea73495eb30086eddfba3caa [id=d9deba004a1a443aa4f51ba4a5ff6891]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.973875 884 tablet_service.cc:1467] Processing CreateTablet for tablet 86908b72e98f4c89ac061f5475e121f4 (DEFAULT_TABLE table=default.Table_d291843c6ba74e52b931a8ce480e81a3 [id=40b1cf93305c406fa7e5db3799538d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.974040 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2d3dfb96c24a0e84e9a7e19ccf195f (DEFAULT_TABLE table=default.Table_4a980fd105b247649ed64a59d33bb362 [id=1260ee8d2f494ff8a675c73966adaf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.974232 894 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0983ecfbec4049a5aa57f1e0b6a106 (DEFAULT_TABLE table=default.Table_910bea2f2ba9433d8ff8f837707b8192 [id=af93c62a9cdd42949d8b2fbcde142749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.974349 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c2d3dfb96c24a0e84e9a7e19ccf195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.974293 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0983ecfbec4049a5aa57f1e0b6a106 (DEFAULT_TABLE table=default.Table_910bea2f2ba9433d8ff8f837707b8192 [id=af93c62a9cdd42949d8b2fbcde142749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.974562 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0983ecfbec4049a5aa57f1e0b6a106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.974650 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3e7bffaac64167aae4974f379b4650 (DEFAULT_TABLE table=default.Table_ed6dd3916ff3440b8f88fb3f3d56f5bf [id=27aceae056004df6a1b7fc88fa5c099d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.974745 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 627a7b307bbd4f1590d5ccd2595a0aaa (DEFAULT_TABLE table=default.Table_8dbd9988a2e0401891d2ae569ca31ffc [id=1a1a1175d0474959a1cf915612e0b31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.974947 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3e7bffaac64167aae4974f379b4650. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.975138 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d4c014521bc04cbb96e52855febc7439 (DEFAULT_TABLE table=default.Table_bdac8179ea73495eb30086eddfba3caa [id=d9deba004a1a443aa4f51ba4a5ff6891]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.975386 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c014521bc04cbb96e52855febc7439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.977690 746 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3e7bffaac64167aae4974f379b4650 (DEFAULT_TABLE table=default.Table_ed6dd3916ff3440b8f88fb3f3d56f5bf [id=27aceae056004df6a1b7fc88fa5c099d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.977962 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3e7bffaac64167aae4974f379b4650. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.980074 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 86908b72e98f4c89ac061f5475e121f4 (DEFAULT_TABLE table=default.Table_d291843c6ba74e52b931a8ce480e81a3 [id=40b1cf93305c406fa7e5db3799538d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.980372 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86908b72e98f4c89ac061f5475e121f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.980901 752 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2d3dfb96c24a0e84e9a7e19ccf195f (DEFAULT_TABLE table=default.Table_4a980fd105b247649ed64a59d33bb362 [id=1260ee8d2f494ff8a675c73966adaf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.981164 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c2d3dfb96c24a0e84e9a7e19ccf195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.984163 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed7abf23ce04a268a460d8e3e97d56b with delete_type TABLET_DATA_DELETED (replaced by tablet 50039da210df47ab8476e28387c8a802) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:09.984792 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a7b307bbd4f1590d5ccd2595a0aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.984948 740 tablet_service.cc:1467] Processing CreateTablet for tablet 627a7b307bbd4f1590d5ccd2595a0aaa (DEFAULT_TABLE table=default.Table_8dbd9988a2e0401891d2ae569ca31ffc [id=1a1a1175d0474959a1cf915612e0b31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.985211 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a7b307bbd4f1590d5ccd2595a0aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.985237 1160 tablet_replica.cc:331] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:09.985397 1160 raft_consensus.cc:2238] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:09.985507 1160 raft_consensus.cc:2267] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:09.986310 1160 ts_tablet_manager.cc:1905] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:09.987020 1157 ts_tablet_manager.cc:1918] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:09.987104 1157 log.cc:1198] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/829bec4f66c24180afd5271057a7d0e8
I20250109 20:53:09.987349 1157 ts_tablet_manager.cc:1939] T 829bec4f66c24180afd5271057a7d0e8 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:09.988472 743 tablet_service.cc:1467] Processing CreateTablet for tablet d4c014521bc04cbb96e52855febc7439 (DEFAULT_TABLE table=default.Table_bdac8179ea73495eb30086eddfba3caa [id=d9deba004a1a443aa4f51ba4a5ff6891]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.988822 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c014521bc04cbb96e52855febc7439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.992905 745 tablet_service.cc:1467] Processing CreateTablet for tablet 86908b72e98f4c89ac061f5475e121f4 (DEFAULT_TABLE table=default.Table_d291843c6ba74e52b931a8ce480e81a3 [id=40b1cf93305c406fa7e5db3799538d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.993155 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86908b72e98f4c89ac061f5475e121f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.994737 1025 tablet_service.cc:1467] Processing CreateTablet for tablet d7e8eec58dce434aaf6aa1b40a4cd18f (DEFAULT_TABLE table=default.Table_b81a2f52069e4ca693327ab785e96d1d [id=e49dc14ae2894fa6844f665ae00178b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.995000 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e8eec58dce434aaf6aa1b40a4cd18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.996923 739 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0983ecfbec4049a5aa57f1e0b6a106 (DEFAULT_TABLE table=default.Table_910bea2f2ba9433d8ff8f837707b8192 [id=af93c62a9cdd42949d8b2fbcde142749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:09.997213 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0983ecfbec4049a5aa57f1e0b6a106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:09.988683 1157 tablet_replica.cc:331] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:09.999907 1157 raft_consensus.cc:2238] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.000021 1157 raft_consensus.cc:2267] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.001092 881 tablet_service.cc:1467] Processing CreateTablet for tablet 81bd8453e72e4eb18ab6cbbc1ba57cc3 (DEFAULT_TABLE table=default.Table_deb4e6bf04ca43d4a1a8dbe6bb09a3da [id=877d2976a99d4a7dadd7fb10a65a7744]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.001493 1157 ts_tablet_manager.cc:1905] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.001502 888 tablet_service.cc:1467] Processing CreateTablet for tablet f9793a3c89164f5c89207c3c151d92c3 (DEFAULT_TABLE table=default.Table_342b51d32c0845ee9786a3b0cd87e309 [id=24f81e16c5e14893b63ea99daf6d3815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.001907 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf9cae8f38f94f4aba1fbe8f7cd4b7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.002048 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9793a3c89164f5c89207c3c151d92c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.005072 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627a7b307bbd4f1590d5ccd2595a0aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.005357 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 829bec4f66c24180afd5271057a7d0e8 (table default.Table_68eba4e7c30d4954a84fbdf550b8cc07 [id=8bd540a812014512abdafc476da1f850]) successfully deleted
W20250109 20:53:10.005546 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet c4cff74250cd435e8a52c010c11eff9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4cff74250cd435e8a52c010c11eff9b already in progress: creating tablet
I20250109 20:53:10.006471 1166 ts_tablet_manager.cc:1918] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.006563 1166 log.cc:1198] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f2383d9770834b1aae290308b4185299
I20250109 20:53:10.006842 1166 ts_tablet_manager.cc:1939] T f2383d9770834b1aae290308b4185299 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.007493 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1479) took 1115 ms. Trace:
I20250109 20:53:10.007557 1166 rpcz_store.cc:276] 0109 20:53:08.892116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.898653 (+ 6537us) service_pool.cc:225] Handling call
0109 20:53:09.940666 (+1042013us) ts_tablet_manager.cc:1145] Deleting tablet f2383d9770834b1aae290308b4185299
0109 20:53:09.940674 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:09.959956 (+ 19282us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:10.006395 (+ 46439us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:10.007476 (+ 1081us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5228,"tablet-delete.queue_time_us":1039698}
I20250109 20:53:10.007892 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f2383d9770834b1aae290308b4185299 (table default.Table_c344526523b540b7ae0c1d1853a60c23 [id=51bb90fe67424c6fa1320fe03540875a]) successfully deleted
I20250109 20:53:10.008448 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3e7bffaac64167aae4974f379b4650. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.008843 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c014521bc04cbb96e52855febc7439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.009194 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c2d3dfb96c24a0e84e9a7e19ccf195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.009364 746 tablet_service.cc:1467] Processing CreateTablet for tablet f9793a3c89164f5c89207c3c151d92c3 (DEFAULT_TABLE table=default.Table_342b51d32c0845ee9786a3b0cd87e309 [id=24f81e16c5e14893b63ea99daf6d3815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.009495 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0983ecfbec4049a5aa57f1e0b6a106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.009660 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9793a3c89164f5c89207c3c151d92c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.014750 1176 tablet_bootstrap.cc:654] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.015591 740 tablet_service.cc:1467] Processing CreateTablet for tablet 81bd8453e72e4eb18ab6cbbc1ba57cc3 (DEFAULT_TABLE table=default.Table_deb4e6bf04ca43d4a1a8dbe6bb09a3da [id=877d2976a99d4a7dadd7fb10a65a7744]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.015892 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81bd8453e72e4eb18ab6cbbc1ba57cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.021481 1158 tablet_bootstrap.cc:492] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.021616 1158 ts_tablet_manager.cc:1397] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.003s
I20250109 20:53:10.022328 1158 raft_consensus.cc:357] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.022476 1158 raft_consensus.cc:383] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.022512 1158 raft_consensus.cc:738] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.022691 1158 consensus_queue.cc:260] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.023097 1158 ts_tablet_manager.cc:1428] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:10.023330 1158 tablet_bootstrap.cc:492] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.023804 1157 ts_tablet_manager.cc:1918] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.023912 1157 log.cc:1198] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/9d7587432d7345e4a225fc7d5c968c27
I20250109 20:53:10.024219 1157 ts_tablet_manager.cc:1939] T 9d7587432d7345e4a225fc7d5c968c27 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:10.026626 1166 tablet_replica.cc:331] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:10.027488 1158 tablet_bootstrap.cc:654] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.028386 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5a464f5de548423a8262bd29df0e27e9 (DEFAULT_TABLE table=default.Table_5d073a5c6ed8485ea6db09fa96c95579 [id=b661f8ea99b842a2830915d8ace929a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.028683 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a464f5de548423a8262bd29df0e27e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.030220 1166 raft_consensus.cc:2238] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.030386 1166 raft_consensus.cc:2267] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.030903 1166 ts_tablet_manager.cc:1905] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.032267 882 tablet_service.cc:1467] Processing CreateTablet for tablet f962d4354e9b46d89e166daa92999583 (DEFAULT_TABLE table=default.Table_74a13a5b1c0e43b99921b064b8c4a82c [id=f8466279fa7f415aa1bc98bb1555eaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.032327 1019 tablet_service.cc:1467] Processing CreateTablet for tablet f9793a3c89164f5c89207c3c151d92c3 (DEFAULT_TABLE table=default.Table_342b51d32c0845ee9786a3b0cd87e309 [id=24f81e16c5e14893b63ea99daf6d3815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.032598 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f962d4354e9b46d89e166daa92999583. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.032642 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9793a3c89164f5c89207c3c151d92c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.032816 1021 tablet_service.cc:1467] Processing CreateTablet for tablet cf591aa4d79d42cca4d53ebe381f0d37 (DEFAULT_TABLE table=default.Table_e9fda031a43448df8b7eb4840379501a [id=de027f88fad54a7d8357aa2bc077926a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.032943 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e2e34d8ac4064d20b2ee511be9a93ccc (DEFAULT_TABLE table=default.Table_01a8703293504b22beea98b21800c92e [id=8775570705454ddcae95ece4fe59e2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.033236 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e34d8ac4064d20b2ee511be9a93ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.033654 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 5f55dd45ad16489cbfaa59338a2fbb44 (DEFAULT_TABLE table=default.Table_c9df47cadc104f3f915d6e77b01d9e1f [id=cf66fc4405fe4039a2f18ba6203a1f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.033820 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 5a464f5de548423a8262bd29df0e27e9 (DEFAULT_TABLE table=default.Table_5d073a5c6ed8485ea6db09fa96c95579 [id=b661f8ea99b842a2830915d8ace929a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.034056 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a464f5de548423a8262bd29df0e27e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.034087 1022 tablet_service.cc:1467] Processing CreateTablet for tablet f962d4354e9b46d89e166daa92999583 (DEFAULT_TABLE table=default.Table_74a13a5b1c0e43b99921b064b8c4a82c [id=f8466279fa7f415aa1bc98bb1555eaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.034076 742 tablet_service.cc:1467] Processing CreateTablet for tablet b71604f6b7f6423a8aa8618989a1f5b8 (DEFAULT_TABLE table=default.Table_e556f33cc6824d0cb0f84e33ae69638e [id=98ccb6262e5c4a1b8e11a821abcdad99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.034339 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f962d4354e9b46d89e166daa92999583. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.034348 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b71604f6b7f6423a8aa8618989a1f5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.034672 1031 tablet_service.cc:1467] Processing CreateTablet for tablet a53ac1da98a64801bb2074dc09c70c28 (DEFAULT_TABLE table=default.Table_1e5cf45918b64420a95b67afccc51b67 [id=cdd73d33a88d4647b1e8c6d7bd8121cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.034909 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53ac1da98a64801bb2074dc09c70c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.034956 1028 tablet_service.cc:1467] Processing CreateTablet for tablet adf78f3887ec437b9cbb06d48eb31c86 (DEFAULT_TABLE table=default.Table_cbc411f1dddc414ca1c4045adc085f38 [id=364291cbafeb408f8a4bbb30c37a1f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.035192 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf78f3887ec437b9cbb06d48eb31c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.035423 1020 tablet_service.cc:1467] Processing CreateTablet for tablet b71604f6b7f6423a8aa8618989a1f5b8 (DEFAULT_TABLE table=default.Table_e556f33cc6824d0cb0f84e33ae69638e [id=98ccb6262e5c4a1b8e11a821abcdad99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.035657 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b71604f6b7f6423a8aa8618989a1f5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.035805 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf591aa4d79d42cca4d53ebe381f0d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.036015 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7ca6c4c5c14685a88c70d270dc625f (DEFAULT_TABLE table=default.Table_819fde99126d4beb8811b3032f6450bb [id=fbb191297f29441b942cffdfc5845590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.036793 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7ca6c4c5c14685a88c70d270dc625f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.037225 755 tablet_service.cc:1467] Processing CreateTablet for tablet 5f55dd45ad16489cbfaa59338a2fbb44 (DEFAULT_TABLE table=default.Table_c9df47cadc104f3f915d6e77b01d9e1f [id=cf66fc4405fe4039a2f18ba6203a1f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.037511 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f55dd45ad16489cbfaa59338a2fbb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.038018 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86908b72e98f4c89ac061f5475e121f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.040961 757 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7ca6c4c5c14685a88c70d270dc625f (DEFAULT_TABLE table=default.Table_819fde99126d4beb8811b3032f6450bb [id=fbb191297f29441b942cffdfc5845590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.041242 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7ca6c4c5c14685a88c70d270dc625f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.043582 885 tablet_service.cc:1467] Processing CreateTablet for tablet adf78f3887ec437b9cbb06d48eb31c86 (DEFAULT_TABLE table=default.Table_cbc411f1dddc414ca1c4045adc085f38 [id=364291cbafeb408f8a4bbb30c37a1f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.043848 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf78f3887ec437b9cbb06d48eb31c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.044919 899 tablet_service.cc:1467] Processing CreateTablet for tablet b71604f6b7f6423a8aa8618989a1f5b8 (DEFAULT_TABLE table=default.Table_e556f33cc6824d0cb0f84e33ae69638e [id=98ccb6262e5c4a1b8e11a821abcdad99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.045056 891 tablet_service.cc:1467] Processing CreateTablet for tablet a53ac1da98a64801bb2074dc09c70c28 (DEFAULT_TABLE table=default.Table_1e5cf45918b64420a95b67afccc51b67 [id=cdd73d33a88d4647b1e8c6d7bd8121cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.045187 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b71604f6b7f6423a8aa8618989a1f5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.045360 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53ac1da98a64801bb2074dc09c70c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.045984 880 tablet_service.cc:1467] Processing CreateTablet for tablet 5f55dd45ad16489cbfaa59338a2fbb44 (DEFAULT_TABLE table=default.Table_c9df47cadc104f3f915d6e77b01d9e1f [id=cf66fc4405fe4039a2f18ba6203a1f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.046260 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f55dd45ad16489cbfaa59338a2fbb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.049005 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f55dd45ad16489cbfaa59338a2fbb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.048942 752 tablet_service.cc:1467] Processing CreateTablet for tablet f962d4354e9b46d89e166daa92999583 (DEFAULT_TABLE table=default.Table_74a13a5b1c0e43b99921b064b8c4a82c [id=f8466279fa7f415aa1bc98bb1555eaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.048961 896 tablet_service.cc:1467] Processing CreateTablet for tablet 3363b70e58ad4a29bd7c3c19ef896008 (DEFAULT_TABLE table=default.Table_78986fd7b4744986aaa94a014f7cc06a [id=154b6bd82b6f4e9aadcd78e73420de75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.049249 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3363b70e58ad4a29bd7c3c19ef896008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.049293 739 tablet_service.cc:1467] Processing CreateTablet for tablet 5a464f5de548423a8262bd29df0e27e9 (DEFAULT_TABLE table=default.Table_5d073a5c6ed8485ea6db09fa96c95579 [id=b661f8ea99b842a2830915d8ace929a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.049805 745 tablet_service.cc:1467] Processing CreateTablet for tablet adf78f3887ec437b9cbb06d48eb31c86 (DEFAULT_TABLE table=default.Table_cbc411f1dddc414ca1c4045adc085f38 [id=364291cbafeb408f8a4bbb30c37a1f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.050088 743 tablet_service.cc:1467] Processing CreateTablet for tablet cf591aa4d79d42cca4d53ebe381f0d37 (DEFAULT_TABLE table=default.Table_e9fda031a43448df8b7eb4840379501a [id=de027f88fad54a7d8357aa2bc077926a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.050379 749 tablet_service.cc:1467] Processing CreateTablet for tablet a53ac1da98a64801bb2074dc09c70c28 (DEFAULT_TABLE table=default.Table_1e5cf45918b64420a95b67afccc51b67 [id=cdd73d33a88d4647b1e8c6d7bd8121cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.050662 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf78f3887ec437b9cbb06d48eb31c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.053090 897 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7ca6c4c5c14685a88c70d270dc625f (DEFAULT_TABLE table=default.Table_819fde99126d4beb8811b3032f6450bb [id=fbb191297f29441b942cffdfc5845590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.053143 2749 raft_consensus.cc:491] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.053337 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7ca6c4c5c14685a88c70d270dc625f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.053252 2749 raft_consensus.cc:513] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.053758 2749 leader_election.cc:290] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:10.053925 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf591aa4d79d42cca4d53ebe381f0d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.054219 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc54e97e91c54afea098624bd85fcc6e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:10.054657 990 leader_election.cc:343] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.056951 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a464f5de548423a8262bd29df0e27e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.057546 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53ac1da98a64801bb2074dc09c70c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.058176 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f962d4354e9b46d89e166daa92999583. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.059382 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 9d7587432d7345e4a225fc7d5c968c27 (table default.Table_b590793190454bb09e3e7dd36515f65a [id=4a57bf26eda04187bd4b854d402f3c19]) successfully deleted
I20250109 20:53:10.059717 746 tablet_service.cc:1467] Processing CreateTablet for tablet e2e34d8ac4064d20b2ee511be9a93ccc (DEFAULT_TABLE table=default.Table_01a8703293504b22beea98b21800c92e [id=8775570705454ddcae95ece4fe59e2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.059983 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e34d8ac4064d20b2ee511be9a93ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.060523 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc54e97e91c54afea098624bd85fcc6e" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.063632 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet ee208bad76734a8fa1ada438bdbbe48c (table default.Table_79fb13be9f0440c9991b67d641ab0b6a [id=238aa3a3a8e74dfba23e60c7d6f049cf]). Expected version 1 got 0
I20250109 20:53:10.064090 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e9289eacd1074dd59e244001042ccbe2" } new_table_name: "default.Table_11400e7f19b94a91902e71ab9b565e90" modify_external_catalogs: true
I20250109 20:53:10.064167 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d476f01f037441babd8bd5d6c4e53bb2" } modify_external_catalogs: true
I20250109 20:53:10.064250 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d476f01f037441babd8bd5d6c4e53bb2" } modify_external_catalogs: true
I20250109 20:53:10.065228 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6e895173440e48faafe5e9bdc37051b5
I20250109 20:53:10.065762 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a2537a166da14ae39e657d27dc555f71" } new_table_name: "default.Table_ecc15f594c404044a7d08b68ef3adaab" modify_external_catalogs: true
W20250109 20:53:10.067303 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c913dc6da1a0401596634d361d025164 consensus state has no leader
I20250109 20:53:10.067679 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1f395e1ff2c84808827b0a02b6575663"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.068161 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f395e1ff2c84808827b0a02b6575663 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.068674 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 81bd8453e72e4eb18ab6cbbc1ba57cc3 (DEFAULT_TABLE table=default.Table_deb4e6bf04ca43d4a1a8dbe6bb09a3da [id=877d2976a99d4a7dadd7fb10a65a7744]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.069052 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81bd8453e72e4eb18ab6cbbc1ba57cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.069346 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet de57e94d00d748dc8b02d60f2f1fb27a
I20250109 20:53:10.069943 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6ee7a6b6b6c142ac8a14a4fe60c7c5b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.070340 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ee7a6b6b6c142ac8a14a4fe60c7c5b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:10.071280 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14e483cd4ae64fe1aa31cb0e4b6d6338 consensus state has no leader
W20250109 20:53:10.071537 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a45772ab0e6d4316a011be39d5316b99 consensus state has no leader
I20250109 20:53:10.072120 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e895173440e48faafe5e9bdc37051b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.072494 889 tablet_service.cc:1514] Processing DeleteTablet for tablet de57e94d00d748dc8b02d60f2f1fb27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:10.072741 989 leader_election.cc:343] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.072837 989 leader_election.cc:304] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.073185 2749 raft_consensus.cc:2743] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.075047 2633 raft_consensus.cc:491] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.075150 2633 raft_consensus.cc:513] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.075673 2633 leader_election.cc:290] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.077535 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8e249bfdf7084d0d838c0c7b47e75609" } modify_external_catalogs: true
I20250109 20:53:10.077654 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8e249bfdf7084d0d838c0c7b47e75609" } modify_external_catalogs: true
I20250109 20:53:10.078954 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65489a602141239c99e8e8dae22ab4" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:10.079558 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65489a602141239c99e8e8dae22ab4" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:10.080018 854 leader_election.cc:304] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:10.080339 2633 raft_consensus.cc:2743] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.080922 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7a9f34b8d72242458260fc2cc645f85f" } modify_external_catalogs: true
I20250109 20:53:10.081051 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7a9f34b8d72242458260fc2cc645f85f" } modify_external_catalogs: true
I20250109 20:53:10.089524 1176 tablet_bootstrap.cc:492] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.089572 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 417aab64aaef487a83c1e0ff6eec9ffc
I20250109 20:53:10.089648 1176 ts_tablet_manager.cc:1397] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.570s user 0.021s sys 0.003s
W20250109 20:53:10.090265 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9e11a8181ae42619a56092017fa9315 consensus state has no leader
I20250109 20:53:10.090353 1176 raft_consensus.cc:357] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.090632 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 30e9398f839f4604819e82794af94351
I20250109 20:53:10.090652 1176 raft_consensus.cc:383] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.090703 1176 raft_consensus.cc:738] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.090893 1176 consensus_queue.cc:260] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:10.091219 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2495dc49929f45dbb55d2e285d0264df consensus state has no leader
I20250109 20:53:10.091578 1176 ts_tablet_manager.cc:1428] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:10.092170 1176 tablet_bootstrap.cc:492] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.093355 1176 tablet_bootstrap.cc:654] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.095407 1165 tablet_bootstrap.cc:654] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.097671 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 417aab64aaef487a83c1e0ff6eec9ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.097683 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e9398f839f4604819e82794af94351 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.104527 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e9398f839f4604819e82794af94351 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:10.104987 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 50afaf6c42a546b2ad7add93095a7556 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50afaf6c42a546b2ad7add93095a7556 already in progress: creating tablet
I20250109 20:53:10.105032 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0e82dbd939465ca7044f0aa1f46f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:10.105167 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 already in progress: creating tablet
I20250109 20:53:10.105342 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e895173440e48faafe5e9bdc37051b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.105506 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet de57e94d00d748dc8b02d60f2f1fb27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.107101 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 417aab64aaef487a83c1e0ff6eec9ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.107592 1166 ts_tablet_manager.cc:1918] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.107687 1166 log.cc:1198] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/7df7405e93aa4718a62933ae56af9f42
I20250109 20:53:10.107828 888 tablet_service.cc:1467] Processing CreateTablet for tablet cf591aa4d79d42cca4d53ebe381f0d37 (DEFAULT_TABLE table=default.Table_e9fda031a43448df8b7eb4840379501a [id=de027f88fad54a7d8357aa2bc077926a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.107949 1166 ts_tablet_manager.cc:1939] T 7df7405e93aa4718a62933ae56af9f42 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.109856 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81bd8453e72e4eb18ab6cbbc1ba57cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.110668 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e895173440e48faafe5e9bdc37051b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.111318 2749 raft_consensus.cc:491] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.111423 2749 raft_consensus.cc:513] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.111884 1158 tablet_bootstrap.cc:492] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.111951 2749 leader_election.cc:290] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:10.111990 1158 ts_tablet_manager.cc:1397] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.000s
I20250109 20:53:10.113612 740 tablet_service.cc:1514] Processing DeleteTablet for tablet de57e94d00d748dc8b02d60f2f1fb27a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.113821 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 417aab64aaef487a83c1e0ff6eec9ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.114480 1176 tablet_bootstrap.cc:492] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.114579 1176 ts_tablet_manager.cc:1397] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250109 20:53:10.115126 1176 raft_consensus.cc:357] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.115273 1176 raft_consensus.cc:383] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.115309 1176 raft_consensus.cc:738] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.115330 450 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet ee208bad76734a8fa1ada438bdbbe48c (table default.Table_79fb13be9f0440c9991b67d641ab0b6a [id=238aa3a3a8e74dfba23e60c7d6f049cf]). Expected version 1 got 0
I20250109 20:53:10.115459 1176 consensus_queue.cc:260] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.116279 1176 ts_tablet_manager.cc:1428] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:10.116851 1165 tablet_bootstrap.cc:492] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:10.116962 1165 ts_tablet_manager.cc:1397] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.156s user 0.004s sys 0.003s
I20250109 20:53:10.117551 1165 raft_consensus.cc:357] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.117689 1165 raft_consensus.cc:383] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.117724 1165 raft_consensus.cc:738] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.117871 1165 consensus_queue.cc:260] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.118263 1165 ts_tablet_manager.cc:1428] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250109 20:53:10.118793 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 34ca0a49aef34c40982ff34464727a5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34ca0a49aef34c40982ff34464727a5e already in progress: creating tablet
I20250109 20:53:10.123286 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa89498a0e554c6b8f69f0c0e407da32" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:10.123576 745 tablet_service.cc:1467] Processing CreateTablet for tablet 3363b70e58ad4a29bd7c3c19ef896008 (DEFAULT_TABLE table=default.Table_78986fd7b4744986aaa94a014f7cc06a [id=154b6bd82b6f4e9aadcd78e73420de75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.123761 891 tablet_service.cc:1467] Processing CreateTablet for tablet 751be9de034b47b497a0d7b35fa10ea7 (DEFAULT_TABLE table=default.Table_8bda12cd6dcd425b96548fb29a430cd9 [id=393564813322418388f7b948eb28801a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.123886 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 751be9de034b47b497a0d7b35fa10ea7 (DEFAULT_TABLE table=default.Table_8bda12cd6dcd425b96548fb29a430cd9 [id=393564813322418388f7b948eb28801a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:10.124097 990 leader_election.cc:343] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.124331 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e9398f839f4604819e82794af94351 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.123886 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 3363b70e58ad4a29bd7c3c19ef896008 (DEFAULT_TABLE table=default.Table_78986fd7b4744986aaa94a014f7cc06a [id=154b6bd82b6f4e9aadcd78e73420de75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.124949 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9517b88c21ab43db8585e7a01615504c (DEFAULT_TABLE table=default.Table_f41293cf3af5440199ff09445139004d [id=151e1acf23f74b6b90c8c66241e669ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.125500 898 tablet_service.cc:1467] Processing CreateTablet for tablet 50e7b1fe60654d6681b7a3c1c1d5dc90 (DEFAULT_TABLE table=default.Table_f54d04dc297c4ff39b8ad462455c0deb [id=d0dc8d3d52344e0a85b4bd8b3afb999d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.125969 742 tablet_service.cc:1467] Processing CreateTablet for tablet 751be9de034b47b497a0d7b35fa10ea7 (DEFAULT_TABLE table=default.Table_8bda12cd6dcd425b96548fb29a430cd9 [id=393564813322418388f7b948eb28801a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.126438 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 9517b88c21ab43db8585e7a01615504c (DEFAULT_TABLE table=default.Table_f41293cf3af5440199ff09445139004d [id=151e1acf23f74b6b90c8c66241e669ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.126847 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 751be9de034b47b497a0d7b35fa10ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.126814 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 50e7b1fe60654d6681b7a3c1c1d5dc90 (DEFAULT_TABLE table=default.Table_f54d04dc297c4ff39b8ad462455c0deb [id=d0dc8d3d52344e0a85b4bd8b3afb999d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.127162 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3363b70e58ad4a29bd7c3c19ef896008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.131384 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 533ac2bcf30d4ff091c5c0d5e418fc70
I20250109 20:53:10.132901 743 tablet_service.cc:1467] Processing CreateTablet for tablet 9517b88c21ab43db8585e7a01615504c (DEFAULT_TABLE table=default.Table_f41293cf3af5440199ff09445139004d [id=151e1acf23f74b6b90c8c66241e669ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.133201 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9517b88c21ab43db8585e7a01615504c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.134048 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 87006af9778543f282e49d83f25beff7 (DEFAULT_TABLE table=default.Table_c6cc9ff227164e74bf9fbc9e799dd5cb [id=47de0100d49a4212ab511df5b420cbba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.134590 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d1dd16809c9a429e8f0a13aadb929df0 (DEFAULT_TABLE table=default.Table_d449e214145248efa415292c5a4a044b [id=20ede1d0f1334e47ac0ac57691ebc8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.134953 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e7b1fe60654d6681b7a3c1c1d5dc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.135305 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1481) took 1166 ms. Trace:
I20250109 20:53:10.135419 1176 tablet_bootstrap.cc:492] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.135385 1166 rpcz_store.cc:276] 0109 20:53:08.969340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.969717 (+ 377us) service_pool.cc:225] Handling call
0109 20:53:10.026573 (+1056856us) ts_tablet_manager.cc:1145] Deleting tablet 7df7405e93aa4718a62933ae56af9f42
0109 20:53:10.026585 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.078046 (+ 51461us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:10.105900 (+ 27854us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:10.135293 (+ 29393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":941690}
I20250109 20:53:10.136085 1157 tablet_replica.cc:331] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:10.136271 1157 raft_consensus.cc:2238] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.136415 1157 raft_consensus.cc:2267] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.136911 1157 ts_tablet_manager.cc:1905] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.137979 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3363b70e58ad4a29bd7c3c19ef896008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.138262 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1482) took 1156 ms. Trace:
I20250109 20:53:10.138325 1166 rpcz_store.cc:276] 0109 20:53:08.981736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.981912 (+ 176us) service_pool.cc:225] Handling call
0109 20:53:10.138163 (+1156251us) ts_tablet_manager.cc:1145] Deleting tablet 07cbc61d839d46d2bae8235d6c92fbbd
0109 20:53:10.138171 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.138246 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053203}
I20250109 20:53:10.138700 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1487) took 1134 ms. Trace:
I20250109 20:53:10.138760 1166 rpcz_store.cc:276] 0109 20:53:09.004233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:09.004303 (+ 70us) service_pool.cc:225] Handling call
0109 20:53:10.138619 (+1134316us) ts_tablet_manager.cc:1145] Deleting tablet fc749eddce6444ee82a1846a19e5c039
0109 20:53:10.138627 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.138688 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049765}
I20250109 20:53:10.139017 1176 tablet_bootstrap.cc:654] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.139127 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1489) took 1122 ms. Trace:
I20250109 20:53:10.139185 1166 rpcz_store.cc:276] 0109 20:53:09.016590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:09.016642 (+ 52us) service_pool.cc:225] Handling call
0109 20:53:10.139048 (+1122406us) ts_tablet_manager.cc:1145] Deleting tablet eae1da4227e54d17bc98ff68f5c36d4f
0109 20:53:10.139056 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.139117 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045251}
I20250109 20:53:10.139605 741 tablet_service.cc:1467] Processing CreateTablet for tablet 50e7b1fe60654d6681b7a3c1c1d5dc90 (DEFAULT_TABLE table=default.Table_f54d04dc297c4ff39b8ad462455c0deb [id=d0dc8d3d52344e0a85b4bd8b3afb999d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.140012 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1483) took 1150 ms. Trace:
I20250109 20:53:10.140067 1166 rpcz_store.cc:276] 0109 20:53:08.989348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:08.989414 (+ 66us) service_pool.cc:225] Handling call
0109 20:53:10.139933 (+1150519us) ts_tablet_manager.cc:1145] Deleting tablet 8a22e629234a4456a63b60017767e67c
0109 20:53:10.139940 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.139996 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045433}
I20250109 20:53:10.140197 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 751be9de034b47b497a0d7b35fa10ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.140376 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1490) took 1012 ms. Trace:
I20250109 20:53:10.140429 1166 rpcz_store.cc:276] 0109 20:53:09.127993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:09.128230 (+ 237us) service_pool.cc:225] Handling call
0109 20:53:10.140305 (+1012075us) ts_tablet_manager.cc:1145] Deleting tablet c4cff74250cd435e8a52c010c11eff9b
0109 20:53:10.140313 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.140366 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006745}
I20250109 20:53:10.140863 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87006af9778543f282e49d83f25beff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.140923 1166 tablet_replica.cc:331] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:10.141125 1166 raft_consensus.cc:2238] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.141240 1166 raft_consensus.cc:2267] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.141604 1166 ts_tablet_manager.cc:1905] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.145210 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1dd16809c9a429e8f0a13aadb929df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.149166 1158 raft_consensus.cc:357] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.149396 1158 raft_consensus.cc:383] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.149456 1158 raft_consensus.cc:738] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.149643 1158 consensus_queue.cc:260] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.150082 1158 ts_tablet_manager.cc:1428] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.038s user 0.001s sys 0.000s
I20250109 20:53:10.150377 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9517b88c21ab43db8585e7a01615504c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.150705 887 tablet_service.cc:1467] Processing CreateTablet for tablet 87006af9778543f282e49d83f25beff7 (DEFAULT_TABLE table=default.Table_c6cc9ff227164e74bf9fbc9e799dd5cb [id=47de0100d49a4212ab511df5b420cbba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.150894 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e7b1fe60654d6681b7a3c1c1d5dc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.151146 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.151423 883 tablet_service.cc:1467] Processing CreateTablet for tablet d1dd16809c9a429e8f0a13aadb929df0 (DEFAULT_TABLE table=default.Table_d449e214145248efa415292c5a4a044b [id=20ede1d0f1334e47ac0ac57691ebc8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.151619 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.151974 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa89498a0e554c6b8f69f0c0e407da32" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.152597 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8ee14609a1444a7f853f33a1c27476cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:10.152655 740 tablet_service.cc:1467] Processing CreateTablet for tablet d1dd16809c9a429e8f0a13aadb929df0 (DEFAULT_TABLE table=default.Table_d449e214145248efa415292c5a4a044b [id=20ede1d0f1334e47ac0ac57691ebc8c8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:10.153059 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ee14609a1444a7f853f33a1c27476cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:10.156239 989 leader_election.cc:343] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.156355 989 leader_election.cc:304] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.156577 2749 raft_consensus.cc:2743] T aa89498a0e554c6b8f69f0c0e407da32 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.158278 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 751be9de034b47b497a0d7b35fa10ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.159006 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3dcc18419938438fbe7c33612df86f1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.159449 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dcc18419938438fbe7c33612df86f1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.159966 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e7b1fe60654d6681b7a3c1c1d5dc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.160976 1165 tablet_bootstrap.cc:492] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:10.163071 1165 tablet_bootstrap.cc:654] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.163913 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9517b88c21ab43db8585e7a01615504c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.164811 739 tablet_service.cc:1467] Processing CreateTablet for tablet 87006af9778543f282e49d83f25beff7 (DEFAULT_TABLE table=default.Table_c6cc9ff227164e74bf9fbc9e799dd5cb [id=47de0100d49a4212ab511df5b420cbba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.165194 1158 tablet_bootstrap.cc:492] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.169256 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cc91766733934db588c5f1153a4b0d0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.171197 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc91766733934db588c5f1153a4b0d0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:10.172263 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 41f0935f17714ecead18b813ccf38f74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41f0935f17714ecead18b813ccf38f74 already in progress: creating tablet
I20250109 20:53:10.172817 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.174386 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 7df7405e93aa4718a62933ae56af9f42 (table default.Table_6223d4bbeb7a4deebddb2936e92224c0 [id=f88d49eba7374387915b697342d8ddd7]) successfully deleted
W20250109 20:53:10.174845 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 07cbc61d839d46d2bae8235d6c92fbbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07cbc61d839d46d2bae8235d6c92fbbd already in progress: creating tablet
W20250109 20:53:10.175119 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet fc749eddce6444ee82a1846a19e5c039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc749eddce6444ee82a1846a19e5c039 already in progress: creating tablet
W20250109 20:53:10.175333 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet eae1da4227e54d17bc98ff68f5c36d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eae1da4227e54d17bc98ff68f5c36d4f already in progress: creating tablet
W20250109 20:53:10.175616 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8a22e629234a4456a63b60017767e67c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a22e629234a4456a63b60017767e67c already in progress: creating tablet
W20250109 20:53:10.175850 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c4cff74250cd435e8a52c010c11eff9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4cff74250cd435e8a52c010c11eff9b already in progress: creating tablet
I20250109 20:53:10.176119 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1dd16809c9a429e8f0a13aadb929df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.181320 1158 tablet_bootstrap.cc:654] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.187093 1166 ts_tablet_manager.cc:1918] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.187216 1166 log.cc:1198] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f8eba3dcab764a98857604b63bf9dce1
I20250109 20:53:10.187525 1166 ts_tablet_manager.cc:1939] T f8eba3dcab764a98857604b63bf9dce1 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.190151 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87006af9778543f282e49d83f25beff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.192658 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 82209a6b39bb48e791ff63a8167c9765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:01 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.192929 1157 ts_tablet_manager.cc:1918] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.193022 1157 log.cc:1198] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/4870a667cd3e410590923c9e2497af54
I20250109 20:53:10.193287 1157 ts_tablet_manager.cc:1939] T 4870a667cd3e410590923c9e2497af54 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
W20250109 20:53:10.193564 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 50afaf6c42a546b2ad7add93095a7556 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50afaf6c42a546b2ad7add93095a7556 already in progress: creating tablet
I20250109 20:53:10.195333 757 tablet_service.cc:1514] Processing DeleteTablet for tablet deedac68322e460b8d62adbe849de3f2 with delete_type TABLET_DATA_DELETED (replaced by tablet be05de223c3946c6b8a4899cf696fe26) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.195911 884 tablet_service.cc:1467] Processing CreateTablet for tablet 01b47d35a305430db0baecd2866b296e (DEFAULT_TABLE table=default.Table_a648ce3f75494fc5bfeb8d6da9685375 [id=1317abc1307248a0ab0d53832a382703]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.198010 1160 ts_tablet_manager.cc:1918] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.198253 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1494) took 1015 ms. Trace:
I20250109 20:53:10.198336 1166 rpcz_store.cc:276] 0109 20:53:09.182594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:09.183469 (+ 875us) service_pool.cc:225] Handling call
0109 20:53:10.140888 (+957419us) ts_tablet_manager.cc:1145] Deleting tablet f8eba3dcab764a98857604b63bf9dce1
0109 20:53:10.140895 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.184101 (+ 43206us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:10.186994 (+ 2893us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:10.198232 (+ 11238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":957193}
I20250109 20:53:10.198565 1160 log.cc:1198] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2ed7abf23ce04a268a460d8e3e97d56b
I20250109 20:53:10.198770 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1495) took 1007 ms. Trace:
I20250109 20:53:10.198843 1166 rpcz_store.cc:276] 0109 20:53:09.190980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:09.191893 (+ 913us) service_pool.cc:225] Handling call
0109 20:53:10.198659 (+1006766us) ts_tablet_manager.cc:1145] Deleting tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99
0109 20:53:10.198668 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.198756 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006620}
I20250109 20:53:10.199223 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37640 (request call id 1496) took 1001 ms. Trace:
I20250109 20:53:10.199294 1166 rpcz_store.cc:276] 0109 20:53:09.197808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:09.197869 (+ 61us) service_pool.cc:225] Handling call
0109 20:53:10.199135 (+1001266us) ts_tablet_manager.cc:1145] Deleting tablet 34ca0a49aef34c40982ff34464727a5e
0109 20:53:10.199143 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:10.199210 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1001080}
I20250109 20:53:10.199458 882 tablet_service.cc:1467] Processing CreateTablet for tablet 51376196a86547c99610385deeb9ffef (DEFAULT_TABLE table=default.Table_4ed34f82b8c04d07b91df48899ffdfc2 [id=04f20922f02943c0bd5e77279a5a6c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.199999 889 tablet_service.cc:1467] Processing CreateTablet for tablet 36d28fe90bbf45178de37e52369aea22 (DEFAULT_TABLE table=default.Table_e340970bd22c4332b51c18771b943166 [id=7b1cc3f406214954b69e020d172f15f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.200054 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 01b47d35a305430db0baecd2866b296e (DEFAULT_TABLE table=default.Table_a648ce3f75494fc5bfeb8d6da9685375 [id=1317abc1307248a0ab0d53832a382703]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.200305 1166 tablet_replica.cc:331] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:10.200444 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b47d35a305430db0baecd2866b296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.200470 1166 raft_consensus.cc:2238] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.200598 1166 raft_consensus.cc:2267] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.200949 742 tablet_service.cc:1467] Processing CreateTablet for tablet 01b47d35a305430db0baecd2866b296e (DEFAULT_TABLE table=default.Table_a648ce3f75494fc5bfeb8d6da9685375 [id=1317abc1307248a0ab0d53832a382703]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.201067 1166 ts_tablet_manager.cc:1905] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.201256 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b47d35a305430db0baecd2866b296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.205461 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f8eba3dcab764a98857604b63bf9dce1 (table default.Table_113e0f4bda2b441986056ee8e3616d94 [id=211bbf02224e419caabe39e274c38b6e]) successfully deleted
I20250109 20:53:10.205863 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 51376196a86547c99610385deeb9ffef (DEFAULT_TABLE table=default.Table_4ed34f82b8c04d07b91df48899ffdfc2 [id=04f20922f02943c0bd5e77279a5a6c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.206346 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51376196a86547c99610385deeb9ffef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.207360 898 tablet_service.cc:1467] Processing CreateTablet for tablet 5136407e809d4b0793da468e787e6231 (DEFAULT_TABLE table=default.Table_78bb824af7494f76a74431aea6b022ff [id=21ec55faf7384fbd8769efcbf72ca395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.208179 891 tablet_service.cc:1467] Processing CreateTablet for tablet d0a5a92a65dc4547881bf914b88c9b84 (DEFAULT_TABLE table=default.Table_2e36aa38ac6b4bf191f0652d3e3b0691 [id=ee78b2245746462599d98cb60a976b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.208287 896 tablet_service.cc:1467] Processing CreateTablet for tablet b008fff5ce674d16b72d5c911523a458 (DEFAULT_TABLE table=default.Table_491d93ee3be24649b2aa425756d6e2de [id=abec494e2b6a45c984529ac07c480c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.209077 897 tablet_service.cc:1467] Processing CreateTablet for tablet e3a462dd2e4c427b8d24b65a2f5973c3 (DEFAULT_TABLE table=default.Table_aac6ddcccf8342c0913ef40e3acccfe0 [id=4d8f8fe8cb574b478577f9f010b54c94]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:10.209230 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1edc8f2b224d1abcbbac5ac8a6fd99 already in progress: creating tablet
I20250109 20:53:10.209429 1157 tablet_replica.cc:331] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:10.209506 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf591aa4d79d42cca4d53ebe381f0d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.209626 1157 raft_consensus.cc:2238] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:10.209707 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a5a92a65dc4547881bf914b88c9b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.209879 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b008fff5ce674d16b72d5c911523a458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.209936 1157 raft_consensus.cc:2267] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:10.210216 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 34ca0a49aef34c40982ff34464727a5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34ca0a49aef34c40982ff34464727a5e already in progress: creating tablet
W20250109 20:53:10.210371 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 41f0935f17714ecead18b813ccf38f74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41f0935f17714ecead18b813ccf38f74 already in progress: creating tablet
W20250109 20:53:10.210506 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2871457499b64529851a4ec12e80838d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2871457499b64529851a4ec12e80838d already in progress: creating tablet
W20250109 20:53:10.210624 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e560e5b490774c288157294ad6f23a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e560e5b490774c288157294ad6f23a66 already in progress: creating tablet
I20250109 20:53:10.210899 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 4870a667cd3e410590923c9e2497af54 (table default.Table_03ed60a072e941b6b5b9a8a477aa9020 [id=1e9c11e6b9294f56bb2efd10bef9af20]) successfully deleted
W20250109 20:53:10.211032 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e560e5b490774c288157294ad6f23a66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e560e5b490774c288157294ad6f23a66 already in progress: creating tablet
W20250109 20:53:10.211149 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2871457499b64529851a4ec12e80838d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2871457499b64529851a4ec12e80838d already in progress: creating tablet
W20250109 20:53:10.211369 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 04b4247f52ce4d33bf49d9625a1592bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04b4247f52ce4d33bf49d9625a1592bf already in progress: creating tablet
I20250109 20:53:10.211923 454 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:10.214438 739 tablet_service.cc:1467] Processing CreateTablet for tablet 51376196a86547c99610385deeb9ffef (DEFAULT_TABLE table=default.Table_4ed34f82b8c04d07b91df48899ffdfc2 [id=04f20922f02943c0bd5e77279a5a6c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.214736 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51376196a86547c99610385deeb9ffef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.215443 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 36d28fe90bbf45178de37e52369aea22 (DEFAULT_TABLE table=default.Table_e340970bd22c4332b51c18771b943166 [id=7b1cc3f406214954b69e020d172f15f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.215737 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36d28fe90bbf45178de37e52369aea22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.217319 2692 raft_consensus.cc:491] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.217417 2692 raft_consensus.cc:513] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.217904 2692 leader_election.cc:290] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.218483 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2769e5906c604bd5b14df79db765ec61" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:10.218914 712 leader_election.cc:343] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.219281 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2769e5906c604bd5b14df79db765ec61" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:10.219432 1053 raft_consensus.cc:2463] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:10.219786 749 tablet_service.cc:1467] Processing CreateTablet for tablet 36d28fe90bbf45178de37e52369aea22 (DEFAULT_TABLE table=default.Table_e340970bd22c4332b51c18771b943166 [id=7b1cc3f406214954b69e020d172f15f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.220149 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36d28fe90bbf45178de37e52369aea22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.221485 740 tablet_service.cc:1467] Processing CreateTablet for tablet 52bd61beee0e473795bcfbb7ac098e51 (DEFAULT_TABLE table=default.Table_0393ed5be2c1415eac8ff6d3ac1c5229 [id=5c4f0df9183b4f91b67afa13d2e94784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.222368 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bd61beee0e473795bcfbb7ac098e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.224538 1166 ts_tablet_manager.cc:1918] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.224629 1166 log.cc:1198] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/37ee437300414eccba9926e824f72f27
I20250109 20:53:10.224901 1166 ts_tablet_manager.cc:1939] T 37ee437300414eccba9926e824f72f27 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.224872 713 leader_election.cc:304] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.225248 2692 raft_consensus.cc:2798] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:10.225313 2692 raft_consensus.cc:491] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:10.225368 2692 raft_consensus.cc:3054] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.227619 2692 raft_consensus.cc:513] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.228118 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5136407e809d4b0793da468e787e6231 (DEFAULT_TABLE table=default.Table_78bb824af7494f76a74431aea6b022ff [id=21ec55faf7384fbd8769efcbf72ca395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.228412 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5136407e809d4b0793da468e787e6231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.228467 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2769e5906c604bd5b14df79db765ec61" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:10.228950 712 leader_election.cc:343] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.229332 2692 leader_election.cc:290] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.229769 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2769e5906c604bd5b14df79db765ec61" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:10.229938 1053 raft_consensus.cc:3054] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.232227 1053 raft_consensus.cc:2463] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:10.232940 713 leader_election.cc:304] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.232950 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 52bd61beee0e473795bcfbb7ac098e51 (DEFAULT_TABLE table=default.Table_0393ed5be2c1415eac8ff6d3ac1c5229 [id=5c4f0df9183b4f91b67afa13d2e94784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.233245 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bd61beee0e473795bcfbb7ac098e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.233336 2692 raft_consensus.cc:2798] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:10.233603 2692 raft_consensus.cc:695] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:10.233875 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 5136407e809d4b0793da468e787e6231 (DEFAULT_TABLE table=default.Table_78bb824af7494f76a74431aea6b022ff [id=21ec55faf7384fbd8769efcbf72ca395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.233850 2692 consensus_queue.cc:237] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.234236 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5136407e809d4b0793da468e787e6231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.235085 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87006af9778543f282e49d83f25beff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.237389 2695 raft_consensus.cc:491] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.237490 1034 tablet_service.cc:1467] Processing CreateTablet for tablet d0a5a92a65dc4547881bf914b88c9b84 (DEFAULT_TABLE table=default.Table_2e36aa38ac6b4bf191f0652d3e3b0691 [id=ee78b2245746462599d98cb60a976b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.237490 2695 raft_consensus.cc:513] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.237769 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a5a92a65dc4547881bf914b88c9b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.238008 2695 leader_election.cc:290] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.238482 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deedac68322e460b8d62adbe849de3f2" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.238832 1166 tablet_replica.cc:331] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:53:10.238873 712 leader_election.cc:343] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.238996 1166 raft_consensus.cc:2238] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.239122 1166 raft_consensus.cc:2267] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.241637 1166 ts_tablet_manager.cc:1905] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.242046 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1dd16809c9a429e8f0a13aadb929df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.243197 757 tablet_service.cc:1467] Processing CreateTablet for tablet b008fff5ce674d16b72d5c911523a458 (DEFAULT_TABLE table=default.Table_491d93ee3be24649b2aa425756d6e2de [id=abec494e2b6a45c984529ac07c480c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.243376 1034 tablet_service.cc:1467] Processing CreateTablet for tablet b008fff5ce674d16b72d5c911523a458 (DEFAULT_TABLE table=default.Table_491d93ee3be24649b2aa425756d6e2de [id=abec494e2b6a45c984529ac07c480c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.243494 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b008fff5ce674d16b72d5c911523a458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.243638 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b008fff5ce674d16b72d5c911523a458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.245904 1157 ts_tablet_manager.cc:1905] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.246235 2695 raft_consensus.cc:491] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.246345 2695 raft_consensus.cc:513] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.246752 1166 ts_tablet_manager.cc:1918] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:53:10.246835 1166 log.cc:1198] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/3911b54a040144baadfc4e0088d58e64
I20250109 20:53:10.246886 2695 leader_election.cc:290] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.247169 1166 ts_tablet_manager.cc:1939] T 3911b54a040144baadfc4e0088d58e64 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.247407 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6cf525cb44c428f7a39a25c4e62dc" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:10.247822 712 leader_election.cc:343] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:10.248090 713 leader_election.cc:343] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: deedac68322e460b8d62adbe849de3f2
I20250109 20:53:10.248175 713 leader_election.cc:304] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:10.248466 2695 raft_consensus.cc:2743] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:10.248886 713 leader_election.cc:343] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 72a6cf525cb44c428f7a39a25c4e62dc
I20250109 20:53:10.248991 713 leader_election.cc:304] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:10.249290 2695 raft_consensus.cc:2743] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.250104 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 37ee437300414eccba9926e824f72f27 (table default.Table_7f957a53ae794811bf294551bf3f6e00 [id=8097eb217f1844ec8e2dc236b2f19256]) successfully deleted
W20250109 20:53:10.250383 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 04b4247f52ce4d33bf49d9625a1592bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04b4247f52ce4d33bf49d9625a1592bf already in progress: creating tablet
I20250109 20:53:10.250576 2695 raft_consensus.cc:491] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.250650 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 3911b54a040144baadfc4e0088d58e64 (table default.Table_1eee2fc76a024644bdd78f7fd1ef0e62 [id=24b47aa4d1f44087a21a07eba9320ab1]) successfully deleted
I20250109 20:53:10.250674 2695 raft_consensus.cc:513] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.251122 2695 leader_election.cc:290] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.251451 742 tablet_service.cc:1467] Processing CreateTablet for tablet d0a5a92a65dc4547881bf914b88c9b84 (DEFAULT_TABLE table=default.Table_2e36aa38ac6b4bf191f0652d3e3b0691 [id=ee78b2245746462599d98cb60a976b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.251668 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b50c91807734e86840e874886dc0749" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:10.252076 712 leader_election.cc:343] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.253088 1166 tablet_replica.cc:331] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:10.253295 1166 raft_consensus.cc:2238] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:10.253307 713 leader_election.cc:343] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 1b50c91807734e86840e874886dc0749
I20250109 20:53:10.253407 713 leader_election.cc:304] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:10.253429 1166 raft_consensus.cc:2267] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.253719 2695 raft_consensus.cc:2743] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.253926 1166 ts_tablet_manager.cc:1905] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.254351 745 tablet_service.cc:1467] Processing CreateTablet for tablet e3a462dd2e4c427b8d24b65a2f5973c3 (DEFAULT_TABLE table=default.Table_aac6ddcccf8342c0913ef40e3acccfe0 [id=4d8f8fe8cb574b478577f9f010b54c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.255316 891 tablet_service.cc:1467] Processing CreateTablet for tablet 42b4324192d1437993be73d94590c8f5 (DEFAULT_TABLE table=default.Table_a6e85bbe71224c05bb4b683d0caa2d57 [id=2cb6e0720db54b2ab0c82408185cf822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.255533 899 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2e7e76894d4ca29d8e3f439cfd750e (DEFAULT_TABLE table=default.Table_e9a53e076bfe41ba92e19b480edff7a5 [id=e36c45f4fd4e496e86c1e847f21dffa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.255632 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42b4324192d1437993be73d94590c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.255824 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2e7e76894d4ca29d8e3f439cfd750e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.255970 912 raft_consensus.cc:3054] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.260293 894 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3d549afa16407482c759588ec1369a (DEFAULT_TABLE table=default.Table_325a0939aec24e8991002b08e86afa13 [id=ed66fdd604214c32a8da1091037898a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.260591 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3d549afa16407482c759588ec1369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.265193 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9a175bd72fd749ac80c63aae2b1567a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.265712 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a175bd72fd749ac80c63aae2b1567a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.266029 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1147e6a8dc904bc498efb7a2dd65c120 (DEFAULT_TABLE table=default.Table_fb5f69358fee4665bc556a1e0feba9c3 [id=45176a49ac644a67860c5e98274d3b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.266342 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1147e6a8dc904bc498efb7a2dd65c120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.266947 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0ac75ea834e740efa59cffbf81f74cb1" } modify_external_catalogs: true
I20250109 20:53:10.267102 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0ac75ea834e740efa59cffbf81f74cb1" } modify_external_catalogs: true
I20250109 20:53:10.269944 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b49ca6bc758a4b8c9c39cc064d3e44c7" } new_table_name: "default.Table_da66d878c7714840ae2e131d245f3c68" modify_external_catalogs: true
I20250109 20:53:10.270303 892 tablet_service.cc:1467] Processing CreateTablet for tablet 07d8d1b1db7c40f3a6272dde5bfa13e8 (DEFAULT_TABLE table=default.Table_4ed44e516e92436080e6b73ac17291c3 [id=e44dbc78ab354aeabfc226d903070dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.270598 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d8d1b1db7c40f3a6272dde5bfa13e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.271723 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6f7d71e007e84fafa7d6634d3ee5e122" } modify_external_catalogs: true
I20250109 20:53:10.271863 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6f7d71e007e84fafa7d6634d3ee5e122" } modify_external_catalogs: true
I20250109 20:53:10.274251 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6636fd244b2448a4a8de2eda78a535a4" } new_table_name: "default.Table_e6ba223109264a98a2a80e20e982b0ba" modify_external_catalogs: true
I20250109 20:53:10.274958 1166 ts_tablet_manager.cc:1918] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.275044 1166 log.cc:1198] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/678ab724db1a42e2b23bbe6cd7652db7
I20250109 20:53:10.275198 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fab4024b118047b49eddc729dc2f7c63" } new_table_name: "default.Table_cc5e0fd132d64de6bcd94f9f4743aa8c" modify_external_catalogs: true
I20250109 20:53:10.275285 1166 ts_tablet_manager.cc:1939] T 678ab724db1a42e2b23bbe6cd7652db7 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.276254 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_df7d6213072043f5b9348690ddce3ceb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:10.276330 880 tablet_service.cc:1467] Processing CreateTablet for tablet 6deb59cbfd634856b0ecd28e55ea77c9 (DEFAULT_TABLE table=default.Table_a0308450aa42436a8a6535ffdeb4a017 [id=155c599082c84b648be3644c69603455]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.276614 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6deb59cbfd634856b0ecd28e55ea77c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:10.276666 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df7d6213072043f5b9348690ddce3ceb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.278693 1176 tablet_bootstrap.cc:492] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.279402 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 6daac072067b4071bf2280f05c47f14b
I20250109 20:53:10.278802 1176 ts_tablet_manager.cc:1397] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.143s user 0.002s sys 0.002s
I20250109 20:53:10.280062 1176 raft_consensus.cc:357] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.280232 1176 raft_consensus.cc:383] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.280280 1176 raft_consensus.cc:738] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.280463 1176 consensus_queue.cc:260] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.280604 886 tablet_service.cc:1467] Processing CreateTablet for tablet a49ae634c5954b95806f7da028b39f23 (DEFAULT_TABLE table=default.Table_7f412afa12c54777ae20a1441128d6de [id=0a52c843ebf749ac9300b95825418e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.280906 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a49ae634c5954b95806f7da028b39f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.264961 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_4800f1d4b4d54e4b9abef2016f1684c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.282578 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4800f1d4b4d54e4b9abef2016f1684c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.283082 1176 ts_tablet_manager.cc:1428] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:10.283330 1176 tablet_bootstrap.cc:492] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.286460 1158 tablet_bootstrap.cc:492] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.286583 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51376196a86547c99610385deeb9ffef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.286597 1158 ts_tablet_manager.cc:1397] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.121s user 0.003s sys 0.007s
I20250109 20:53:10.287281 1158 raft_consensus.cc:357] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.287441 1158 raft_consensus.cc:383] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.287482 1158 raft_consensus.cc:738] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:10.288554 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d127e2502b1547189e37e4b8f0dddc1f consensus state has no leader
I20250109 20:53:10.288913 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet af77e079fe5642d790707406c5588420
I20250109 20:53:10.287653 1158 consensus_queue.cc:260] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.289786 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d6579e9bf91a4d5a93b219c3e20784ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:10.289947 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a462dd2e4c427b8d24b65a2f5973c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.290180 1158 ts_tablet_manager.cc:1428] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250109 20:53:10.290215 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6579e9bf91a4d5a93b219c3e20784ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.290413 1158 tablet_bootstrap.cc:492] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.294683 885 tablet_service.cc:1467] Processing CreateTablet for tablet e620635f69b84608b44fca9649738c3b (DEFAULT_TABLE table=default.Table_1359ff2be59e4f6683ac7b4577a85027 [id=442ee55f2e6e45d8918d5016d7d1f873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.294346 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4033dea6a18347b4b0eaf9022ceb6402" } modify_external_catalogs: true
I20250109 20:53:10.294921 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4033dea6a18347b4b0eaf9022ceb6402" } modify_external_catalogs: true
I20250109 20:53:10.295213 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e620635f69b84608b44fca9649738c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.294456 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c75db1b93cbd4edb9fce068615c63752" } new_table_name: "default.Table_82cfa4b004c3462394b991007b7140d0" modify_external_catalogs: true
I20250109 20:53:10.298023 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5136407e809d4b0793da468e787e6231. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:10.301393 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 769bee026cfc4438b78f46ce91bd1b9d has no consensus state
I20250109 20:53:10.294608 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b652be7142be44bab45b5ad83a328a06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.302654 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b652be7142be44bab45b5ad83a328a06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.303380 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b47d35a305430db0baecd2866b296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.303757 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_eaba7b7ff5d44921a1ce8461ba178fbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.304149 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaba7b7ff5d44921a1ce8461ba178fbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.294211 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d31a1416d67e49c58ce51ad2c7a4f6c7" } modify_external_catalogs: true
I20250109 20:53:10.305327 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d31a1416d67e49c58ce51ad2c7a4f6c7" } modify_external_catalogs: true
I20250109 20:53:10.294728 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9c8540e2571f47ababd75f8794ea8134" } new_table_name: "default.Table_b4269e45a82a447bb9575e06acd6bb1a" modify_external_catalogs: true
I20250109 20:53:10.307583 1157 ts_tablet_manager.cc:1918] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:53:10.307686 1157 log.cc:1198] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/3911b54a040144baadfc4e0088d58e64
I20250109 20:53:10.308022 1157 ts_tablet_manager.cc:1939] T 3911b54a040144baadfc4e0088d58e64 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:10.308480 1166 tablet_replica.cc:331] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:10.308655 1166 raft_consensus.cc:2238] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.308691 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36d28fe90bbf45178de37e52369aea22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.308662 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a56fbc8f36a2420a992dea0a8126723a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.309130 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a56fbc8f36a2420a992dea0a8126723a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.313167 1166 raft_consensus.cc:2267] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.313603 1158 tablet_bootstrap.cc:654] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.313951 1176 tablet_bootstrap.cc:654] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.293948 1034 tablet_service.cc:1467] Processing CreateTablet for tablet e3a462dd2e4c427b8d24b65a2f5973c3 (DEFAULT_TABLE table=default.Table_aac6ddcccf8342c0913ef40e3acccfe0 [id=4d8f8fe8cb574b478577f9f010b54c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.314749 1165 tablet_bootstrap.cc:492] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:10.314889 1165 ts_tablet_manager.cc:1397] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.154s user 0.005s sys 0.007s
I20250109 20:53:10.315022 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a462dd2e4c427b8d24b65a2f5973c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.315654 1165 raft_consensus.cc:357] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.316534 1165 raft_consensus.cc:383] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.316599 1165 raft_consensus.cc:738] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.316810 1165 consensus_queue.cc:260] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.317340 1165 ts_tablet_manager.cc:1428] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:10.317551 1165 tablet_bootstrap.cc:492] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:10.319944 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 1147e6a8dc904bc498efb7a2dd65c120 (DEFAULT_TABLE table=default.Table_fb5f69358fee4665bc556a1e0feba9c3 [id=45176a49ac644a67860c5e98274d3b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.320290 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1147e6a8dc904bc498efb7a2dd65c120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.319375 757 tablet_service.cc:1467] Processing CreateTablet for tablet 42b4324192d1437993be73d94590c8f5 (DEFAULT_TABLE table=default.Table_a6e85bbe71224c05bb4b683d0caa2d57 [id=2cb6e0720db54b2ab0c82408185cf822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.321328 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42b4324192d1437993be73d94590c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:10.323390 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 64273.5us
I20250109 20:53:10.325824 1166 ts_tablet_manager.cc:1905] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:10.327971 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e19d64024eb94df69b2c5d52791bb4c6 consensus state has no leader
I20250109 20:53:10.328545 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5c8d8d771b56407d94fcb23d76796039
I20250109 20:53:10.328684 1157 tablet_replica.cc:331] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:10.328903 1157 raft_consensus.cc:2238] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.329026 1157 raft_consensus.cc:2267] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.329499 1157 ts_tablet_manager.cc:1905] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:10.331295 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3df9ce1c679145179722f6c8d0436e03 consensus state has no leader
I20250109 20:53:10.332795 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet a58b1be0912642c98e566e03a607c1e5
I20250109 20:53:10.333029 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a5a92a65dc4547881bf914b88c9b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.333431 741 tablet_service.cc:1467] Processing CreateTablet for tablet 1147e6a8dc904bc498efb7a2dd65c120 (DEFAULT_TABLE table=default.Table_fb5f69358fee4665bc556a1e0feba9c3 [id=45176a49ac644a67860c5e98274d3b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.334681 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_da561625a5f54a3c9de94e91dbc12c0d" } modify_external_catalogs: true
I20250109 20:53:10.334831 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_da561625a5f54a3c9de94e91dbc12c0d" } modify_external_catalogs: true
I20250109 20:53:10.336064 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3a462dd2e4c427b8d24b65a2f5973c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.340026 1158 tablet_bootstrap.cc:492] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.340145 1158 ts_tablet_manager.cc:1397] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20250109 20:53:10.340751 1158 raft_consensus.cc:357] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.340921 1158 raft_consensus.cc:383] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.340968 1158 raft_consensus.cc:738] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.341027 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 42b4324192d1437993be73d94590c8f5 (DEFAULT_TABLE table=default.Table_a6e85bbe71224c05bb4b683d0caa2d57 [id=2cb6e0720db54b2ab0c82408185cf822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.341153 1158 consensus_queue.cc:260] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.341611 1158 ts_tablet_manager.cc:1428] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:10.341857 1158 tablet_bootstrap.cc:492] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.342734 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5f7fc8c0a0724a598d56532587cf8366
I20250109 20:53:10.343039 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1147e6a8dc904bc498efb7a2dd65c120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.343143 1158 tablet_bootstrap.cc:654] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.348534 1165 tablet_bootstrap.cc:654] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.349841 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2e7e76894d4ca29d8e3f439cfd750e (DEFAULT_TABLE table=default.Table_e9a53e076bfe41ba92e19b480edff7a5 [id=e36c45f4fd4e496e86c1e847f21dffa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.351013 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 07d8d1b1db7c40f3a6272dde5bfa13e8 (DEFAULT_TABLE table=default.Table_4ed44e516e92436080e6b73ac17291c3 [id=e44dbc78ab354aeabfc226d903070dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.351341 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6deb59cbfd634856b0ecd28e55ea77c9 (DEFAULT_TABLE table=default.Table_a0308450aa42436a8a6535ffdeb4a017 [id=155c599082c84b648be3644c69603455]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.351675 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6deb59cbfd634856b0ecd28e55ea77c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.357098 453 master_service.cc:946] ReplaceTablet: received request to replace tablet b1e6beb38a994f02a6c1d7357c2dbd63 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:10.359390 1157 ts_tablet_manager.cc:1918] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.359508 1157 log.cc:1198] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/fce062dc0ac144a19f8fa3eef4a099b4
I20250109 20:53:10.359778 1157 ts_tablet_manager.cc:1939] T fce062dc0ac144a19f8fa3eef4a099b4 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:10.360620 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42b4324192d1437993be73d94590c8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.360886 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2e7e76894d4ca29d8e3f439cfd750e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.362090 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d8d1b1db7c40f3a6272dde5bfa13e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.362242 1158 tablet_bootstrap.cc:492] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.362403 1158 ts_tablet_manager.cc:1397] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20250109 20:53:10.363111 1158 raft_consensus.cc:357] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.363313 1158 raft_consensus.cc:383] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.363376 1158 raft_consensus.cc:738] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.363557 1158 consensus_queue.cc:260] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.364266 1158 ts_tablet_manager.cc:1428] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:10.364571 1158 tablet_bootstrap.cc:492] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.365993 1158 tablet_bootstrap.cc:654] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.367331 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet b1e6beb38a994f02a6c1d7357c2dbd63
I20250109 20:53:10.367892 453 catalog_manager.cc:6596] ReplaceTablet: tablet b1e6beb38a994f02a6c1d7357c2dbd63 deleted and replaced by tablet 97e0335cef0c432fbe2a4002e9876c52
I20250109 20:53:10.368570 881 tablet_service.cc:1467] Processing CreateTablet for tablet 52bd61beee0e473795bcfbb7ac098e51 (DEFAULT_TABLE table=default.Table_0393ed5be2c1415eac8ff6d3ac1c5229 [id=5c4f0df9183b4f91b67afa13d2e94784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.368630 740 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3d549afa16407482c759588ec1369a (DEFAULT_TABLE table=default.Table_325a0939aec24e8991002b08e86afa13 [id=ed66fdd604214c32a8da1091037898a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.369025 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3d549afa16407482c759588ec1369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.369279 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e6beb38a994f02a6c1d7357c2dbd63 with delete_type TABLET_DATA_DELETED (replaced by tablet 97e0335cef0c432fbe2a4002e9876c52) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.370925 1166 ts_tablet_manager.cc:1918] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.371037 1166 log.cc:1198] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/1b0e82dbd939465ca7044f0aa1f46f06
I20250109 20:53:10.371282 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2e7e76894d4ca29d8e3f439cfd750e (DEFAULT_TABLE table=default.Table_e9a53e076bfe41ba92e19b480edff7a5 [id=e36c45f4fd4e496e86c1e847f21dffa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.371357 1166 ts_tablet_manager.cc:1939] T 1b0e82dbd939465ca7044f0aa1f46f06 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.371397 2749 raft_consensus.cc:491] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.371515 2749 raft_consensus.cc:513] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.371732 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2e7e76894d4ca29d8e3f439cfd750e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.372103 2749 leader_election.cc:290] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:10.373126 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45cd5ffe7040e385d8cdfb97ed011a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:10.373589 990 leader_election.cc:343] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.373890 1165 tablet_bootstrap.cc:492] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:10.374001 1165 ts_tablet_manager.cc:1397] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20250109 20:53:10.374548 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45cd5ffe7040e385d8cdfb97ed011a" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.374766 1165 raft_consensus.cc:357] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.374928 1165 raft_consensus.cc:383] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.374971 1165 raft_consensus.cc:738] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:53:10.374976 989 leader_election.cc:343] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.375062 989 leader_election.cc:304] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.375123 1165 consensus_queue.cc:260] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.375412 2749 raft_consensus.cc:2743] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.375558 1165 ts_tablet_manager.cc:1428] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:10.375672 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2883eb1cae054dc4b3073c4944e5355f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.376452 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2883eb1cae054dc4b3073c4944e5355f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.376585 895 tablet_service.cc:1467] Processing CreateTablet for tablet e2e34d8ac4064d20b2ee511be9a93ccc (DEFAULT_TABLE table=default.Table_01a8703293504b22beea98b21800c92e [id=8775570705454ddcae95ece4fe59e2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.376936 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e34d8ac4064d20b2ee511be9a93ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.378823 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d1cde7e6cd224450a38f47398b23e38c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.379231 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1cde7e6cd224450a38f47398b23e38c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.380975 742 tablet_service.cc:1467] Processing CreateTablet for tablet 07d8d1b1db7c40f3a6272dde5bfa13e8 (DEFAULT_TABLE table=default.Table_4ed44e516e92436080e6b73ac17291c3 [id=e44dbc78ab354aeabfc226d903070dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.381350 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d8d1b1db7c40f3a6272dde5bfa13e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.382360 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e620635f69b84608b44fca9649738c3b (DEFAULT_TABLE table=default.Table_1359ff2be59e4f6683ac7b4577a85027 [id=442ee55f2e6e45d8918d5016d7d1f873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.382683 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e620635f69b84608b44fca9649738c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.385435 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e2c554ed27a3434b8133da5863d3ec1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:10.385764 1160 ts_tablet_manager.cc:1939] T 2ed7abf23ce04a268a460d8e3e97d56b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:53:10.385871 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2c554ed27a3434b8133da5863d3ec1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.386412 1157 tablet_replica.cc:331] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:10.387048 1160 tablet_replica.cc:331] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:10.387280 1160 raft_consensus.cc:2238] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.387476 1160 raft_consensus.cc:2267] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.387892 1157 raft_consensus.cc:2238] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.388056 1157 raft_consensus.cc:2267] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.388373 1160 ts_tablet_manager.cc:1905] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.388557 1158 tablet_bootstrap.cc:492] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.388679 1158 ts_tablet_manager.cc:1397] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250109 20:53:10.389283 1157 ts_tablet_manager.cc:1905] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.389521 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2ed7abf23ce04a268a460d8e3e97d56b (table default.Table_422152ed650445aeb8f7af679f9c1d12 [id=b296e8173f88419a8a14ac395611b4bc]) successfully deleted
W20250109 20:53:10.389736 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a45772ab0e6d4316a011be39d5316b99 (table default.Table_07fa5aac1a9d4b4ba13232ba3e4f3571 [id=e2db90856a6f4baa8893fa127e65bf74], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a45772ab0e6d4316a011be39d5316b99 consensus state has no leader
W20250109 20:53:10.390097 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6e895173440e48faafe5e9bdc37051b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e895173440e48faafe5e9bdc37051b5 already in progress: creating tablet
W20250109 20:53:10.390615 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet de57e94d00d748dc8b02d60f2f1fb27a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de57e94d00d748dc8b02d60f2f1fb27a already in progress: creating tablet
W20250109 20:53:10.390796 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 417aab64aaef487a83c1e0ff6eec9ffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417aab64aaef487a83c1e0ff6eec9ffc already in progress: creating tablet
W20250109 20:53:10.390962 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 30e9398f839f4604819e82794af94351 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30e9398f839f4604819e82794af94351 already in progress: creating tablet
W20250109 20:53:10.391176 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533ac2bcf30d4ff091c5c0d5e418fc70 already in progress: creating tablet
I20250109 20:53:10.394614 752 tablet_service.cc:1467] Processing CreateTablet for tablet a49ae634c5954b95806f7da028b39f23 (DEFAULT_TABLE table=default.Table_7f412afa12c54777ae20a1441128d6de [id=0a52c843ebf749ac9300b95825418e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.394989 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a49ae634c5954b95806f7da028b39f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.396217 1158 raft_consensus.cc:357] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.396399 1158 raft_consensus.cc:383] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.396469 1158 raft_consensus.cc:738] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.396687 1158 consensus_queue.cc:260] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.397207 1158 ts_tablet_manager.cc:1428] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250109 20:53:10.397426 1158 tablet_bootstrap.cc:492] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.397439 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bd61beee0e473795bcfbb7ac098e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.402261 1029 tablet_service.cc:1467] Processing CreateTablet for tablet a49ae634c5954b95806f7da028b39f23 (DEFAULT_TABLE table=default.Table_7f412afa12c54777ae20a1441128d6de [id=0a52c843ebf749ac9300b95825418e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.402565 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a49ae634c5954b95806f7da028b39f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.408674 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 6daac072067b4071bf2280f05c47f14b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.409006 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 6daac072067b4071bf2280f05c47f14b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.409173 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet faeec3dbcaff4de98e94357f20e7e8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.409284 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 7a718ccdef3546d5beb8de11b9b9f7c8 (DEFAULT_TABLE table=default.Table_ed0986b3b49e4efbb806df16467e6164 [id=09dabe01772b4d17b2ca71247f62e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.409649 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a718ccdef3546d5beb8de11b9b9f7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.409651 743 tablet_service.cc:1514] Processing DeleteTablet for tablet af77e079fe5642d790707406c5588420 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.409828 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet af77e079fe5642d790707406c5588420 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.409933 1166 tablet_replica.cc:331] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:10.410213 1166 raft_consensus.cc:2238] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.410426 1166 raft_consensus.cc:2267] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.410599 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 3911b54a040144baadfc4e0088d58e64 (table default.Table_1eee2fc76a024644bdd78f7fd1ef0e62 [id=24b47aa4d1f44087a21a07eba9320ab1]) successfully deleted
I20250109 20:53:10.411301 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet fce062dc0ac144a19f8fa3eef4a099b4 (table default.Table_986f5a94df1245f7930aca6efd8614a2 [id=2600eebbccdb4d098f1b48223a9edcd7]) successfully deleted
W20250109 20:53:10.411641 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet caf806685f914510b53c076f1cfd6bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caf806685f914510b53c076f1cfd6bc5 already in progress: creating tablet
W20250109 20:53:10.411826 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8de735008fec4af0a7181cc10426842c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8de735008fec4af0a7181cc10426842c already in progress: creating tablet
W20250109 20:53:10.412086 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet caf806685f914510b53c076f1cfd6bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caf806685f914510b53c076f1cfd6bc5 already in progress: creating tablet
I20250109 20:53:10.412230 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 678ab724db1a42e2b23bbe6cd7652db7 (table default.Table_546faac34e4d4250b0845e4886d01377 [id=db91cf220f2d4191832b2c3c6cddf99d]) successfully deleted
W20250109 20:53:10.412417 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 35fc6c2816a342859e07a52244978186 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35fc6c2816a342859e07a52244978186 already in progress: creating tablet
W20250109 20:53:10.412617 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8de735008fec4af0a7181cc10426842c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8de735008fec4af0a7181cc10426842c already in progress: creating tablet
W20250109 20:53:10.412953 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5d841529b90145caad4c4fd19f9c9669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d841529b90145caad4c4fd19f9c9669 already in progress: creating tablet
W20250109 20:53:10.413458 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1181f2e015334972914ac266d51ac537 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1181f2e015334972914ac266d51ac537 already in progress: creating tablet
W20250109 20:53:10.413640 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2a8e0351fe654fa68e31e4308beb1c79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a8e0351fe654fa68e31e4308beb1c79 already in progress: creating tablet
I20250109 20:53:10.413978 746 tablet_service.cc:1467] Processing CreateTablet for tablet e620635f69b84608b44fca9649738c3b (DEFAULT_TABLE table=default.Table_1359ff2be59e4f6683ac7b4577a85027 [id=442ee55f2e6e45d8918d5016d7d1f873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.414355 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e620635f69b84608b44fca9649738c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.416895 894 tablet_service.cc:1514] Processing DeleteTablet for tablet af77e079fe5642d790707406c5588420 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.427621 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3d549afa16407482c759588ec1369a (DEFAULT_TABLE table=default.Table_325a0939aec24e8991002b08e86afa13 [id=ed66fdd604214c32a8da1091037898a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.428066 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3d549afa16407482c759588ec1369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.428257 1158 tablet_bootstrap.cc:654] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.431017 1157 ts_tablet_manager.cc:1918] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.431118 1157 log.cc:1198] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/ea16fcc30f2c4b2eac53f20fcebd043f
I20250109 20:53:10.431419 1157 ts_tablet_manager.cc:1939] T ea16fcc30f2c4b2eac53f20fcebd043f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:10.434757 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8d8d771b56407d94fcb23d76796039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.435539 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 6daac072067b4071bf2280f05c47f14b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.437135 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7a718ccdef3546d5beb8de11b9b9f7c8 (DEFAULT_TABLE table=default.Table_ed0986b3b49e4efbb806df16467e6164 [id=09dabe01772b4d17b2ca71247f62e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.437232 1165 tablet_bootstrap.cc:492] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:10.437551 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a718ccdef3546d5beb8de11b9b9f7c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.439079 1165 tablet_bootstrap.cc:654] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.443584 898 tablet_service.cc:1467] Processing CreateTablet for tablet 7a718ccdef3546d5beb8de11b9b9f7c8 (DEFAULT_TABLE table=default.Table_ed0986b3b49e4efbb806df16467e6164 [id=09dabe01772b4d17b2ca71247f62e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.443903 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a718ccdef3546d5beb8de11b9b9f7c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:10.446348 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 012ce4d7dbe14a0795497fd3be2f9feb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012ce4d7dbe14a0795497fd3be2f9feb already in progress: creating tablet
W20250109 20:53:10.446659 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 30e9398f839f4604819e82794af94351 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30e9398f839f4604819e82794af94351 already in progress: creating tablet
I20250109 20:53:10.447152 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 1b0e82dbd939465ca7044f0aa1f46f06 (table default.Table_a2b81fc56f9c469e9b72ed20afcb7e02 [id=b70cbd7bddf9479c83c7def3f4b819cf]) successfully deleted
W20250109 20:53:10.447355 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6e895173440e48faafe5e9bdc37051b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e895173440e48faafe5e9bdc37051b5 already in progress: creating tablet
W20250109 20:53:10.447583 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet de57e94d00d748dc8b02d60f2f1fb27a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de57e94d00d748dc8b02d60f2f1fb27a already in progress: creating tablet
W20250109 20:53:10.447793 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 417aab64aaef487a83c1e0ff6eec9ffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417aab64aaef487a83c1e0ff6eec9ffc already in progress: creating tablet
W20250109 20:53:10.447988 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533ac2bcf30d4ff091c5c0d5e418fc70 already in progress: creating tablet
W20250109 20:53:10.448256 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 6daac072067b4071bf2280f05c47f14b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6daac072067b4071bf2280f05c47f14b already in progress: creating tablet
I20250109 20:53:10.449817 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e6beb38a994f02a6c1d7357c2dbd63 with delete_type TABLET_DATA_DELETED (replaced by tablet 97e0335cef0c432fbe2a4002e9876c52) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.451546 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8d8d771b56407d94fcb23d76796039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.451728 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8d8d771b56407d94fcb23d76796039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.453001 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 6deb59cbfd634856b0ecd28e55ea77c9 (DEFAULT_TABLE table=default.Table_a0308450aa42436a8a6535ffdeb4a017 [id=155c599082c84b648be3644c69603455]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.453361 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6deb59cbfd634856b0ecd28e55ea77c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.456804 2784 tablet.cc:1721] T 1ce64e0f3a004c9387bb592cf19236e7 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:10.457486 2785 tablet.cc:1721] T 1ce64e0f3a004c9387bb592cf19236e7 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:10.458714 1165 tablet_bootstrap.cc:492] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:10.458887 1165 ts_tablet_manager.cc:1397] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20250109 20:53:10.460134 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet ea16fcc30f2c4b2eac53f20fcebd043f (table default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d [id=35315fe696f042ee8700dd905bce5c5e]) successfully deleted
W20250109 20:53:10.464915 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5d841529b90145caad4c4fd19f9c9669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d841529b90145caad4c4fd19f9c9669 already in progress: creating tablet
W20250109 20:53:10.465202 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 35fc6c2816a342859e07a52244978186 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35fc6c2816a342859e07a52244978186 already in progress: creating tablet
I20250109 20:53:10.470242 757 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e6beb38a994f02a6c1d7357c2dbd63 with delete_type TABLET_DATA_DELETED (replaced by tablet 97e0335cef0c432fbe2a4002e9876c52) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.479362 1176 tablet_bootstrap.cc:492] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.479488 1176 ts_tablet_manager.cc:1397] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.196s user 0.013s sys 0.017s
I20250109 20:53:10.480042 1176 raft_consensus.cc:357] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.480211 1176 raft_consensus.cc:383] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.480254 1176 raft_consensus.cc:738] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.480418 1176 consensus_queue.cc:260] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.480969 2787 tablet.cc:1721] T 1ce64e0f3a004c9387bb592cf19236e7 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:10.481585 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4c590bc0e47c49328345d4cee4477b02" } modify_external_catalogs: true
I20250109 20:53:10.481745 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4c590bc0e47c49328345d4cee4477b02" } modify_external_catalogs: true
I20250109 20:53:10.482030 1158 tablet_bootstrap.cc:492] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.482149 1158 ts_tablet_manager.cc:1397] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.001s
I20250109 20:53:10.482758 1158 raft_consensus.cc:357] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.482918 1158 raft_consensus.cc:383] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.482960 1158 raft_consensus.cc:738] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.483258 1158 consensus_queue.cc:260] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.483707 1158 ts_tablet_manager.cc:1428] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:10.483707 1176 ts_tablet_manager.cc:1428] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:10.485741 2785 tablet.cc:1721] T 1c65489a602141239c99e8e8dae22ab4 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:10.486786 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_85817f72c90941a2af0506a8759f34c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.487334 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85817f72c90941a2af0506a8759f34c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.490376 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet e274320fcf7c4f2687fad7b8c90909ce
I20250109 20:53:10.490900 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_21df3bf9edb24336a97d09e581e2548b" } new_table_name: "default.Table_0c308430d7a940089dbd71816409aac6" modify_external_catalogs: true
I20250109 20:53:10.491915 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_96de890f59cf48cb8d0b23bb9d8063fe" } modify_external_catalogs: true
I20250109 20:53:10.492069 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_96de890f59cf48cb8d0b23bb9d8063fe" } modify_external_catalogs: true
W20250109 20:53:10.501770 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b412545bb9f4a6a841be1dce4e28c37 consensus state has no leader
I20250109 20:53:10.502367 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c542e77265f247738f6d277b304ffcbd" } new_table_name: "default.Table_ce339fb5da804bea8cf86e50aff0abba" modify_external_catalogs: true
I20250109 20:53:10.503422 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_23347d9f3f95412a95ad830c547d052e" } new_table_name: "default.Table_a7b76d111d3046d8890bc96cfe73deeb" modify_external_catalogs: true
I20250109 20:53:10.504345 1166 ts_tablet_manager.cc:1905] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.505795 1160 ts_tablet_manager.cc:1918] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.505932 1160 log.cc:1198] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/82209a6b39bb48e791ff63a8167c9765
I20250109 20:53:10.506232 1160 ts_tablet_manager.cc:1939] T 82209a6b39bb48e791ff63a8167c9765 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:10.506924 1157 tablet_replica.cc:331] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:10.507118 1157 raft_consensus.cc:2238] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.507272 1157 raft_consensus.cc:2267] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.507758 1157 ts_tablet_manager.cc:1905] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:10.507798 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1181f2e015334972914ac266d51ac537 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1181f2e015334972914ac266d51ac537 already in progress: creating tablet
W20250109 20:53:10.508018 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2a8e0351fe654fa68e31e4308beb1c79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a8e0351fe654fa68e31e4308beb1c79 already in progress: creating tablet
I20250109 20:53:10.508308 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_507ede4faf0d4c83b5844d0247715180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.508782 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_507ede4faf0d4c83b5844d0247715180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.514132 1157 ts_tablet_manager.cc:1918] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.514240 1157 log.cc:1198] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/1b50c91807734e86840e874886dc0749
I20250109 20:53:10.514496 1157 ts_tablet_manager.cc:1939] T 1b50c91807734e86840e874886dc0749 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:10.514575 453 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 1c65489a602141239c99e8e8dae22ab4 (table default.Table_4a9c362f5f8849d88cadca17da9fffe3 [id=2d996a757dae433a9853245f2ceb1f4f]). Expected version 4 got 0
I20250109 20:53:10.514889 1158 tablet_bootstrap.cc:492] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.516431 1158 tablet_bootstrap.cc:654] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.516505 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c0f80ec2094da0893fb0a92223b1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:10.516947 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3846effc51964fd88925fa45cbb3c22b has no consensus state
W20250109 20:53:10.518407 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet caa4c29aff0f4914b8495c83d952c3a6 consensus state has no leader
I20250109 20:53:10.519009 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 1b50c91807734e86840e874886dc0749 (table default.Table_189b7740a7c34ecbbec972fa47180efe [id=902f745269374ba6a0c4ecbeeab62bf9]) successfully deleted
W20250109 20:53:10.520501 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 012ce4d7dbe14a0795497fd3be2f9feb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012ce4d7dbe14a0795497fd3be2f9feb already in progress: creating tablet
I20250109 20:53:10.520910 1157 tablet_replica.cc:331] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:10.521077 1157 raft_consensus.cc:2238] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.521214 1157 raft_consensus.cc:2267] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.521230 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0d846b274cdc42229ed64d25124b2560"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.521663 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d846b274cdc42229ed64d25124b2560 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.523169 1157 ts_tablet_manager.cc:1905] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.523885 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 2ddb127ac2ad49b9820a91a0f6b010a8 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:10.530500 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e8e4034056dd4856a0bdb4c83d34b6d9" } modify_external_catalogs: true
I20250109 20:53:10.530648 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e8e4034056dd4856a0bdb4c83d34b6d9" } modify_external_catalogs: true
I20250109 20:53:10.532750 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0c3b555d74764dc09f834d3ffb88e038"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.533195 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c3b555d74764dc09f834d3ffb88e038 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.534042 2749 raft_consensus.cc:491] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.534157 2749 raft_consensus.cc:513] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.534204 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ef211780e3a44c84b59dca475eb0006e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.534566 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef211780e3a44c84b59dca475eb0006e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.534698 2749 leader_election.cc:290] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:10.535247 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f4bdda16554d9d81de5ca5c5205f4c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.535450 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_261f03425c0e4e77873973fa3746a8c3" } new_table_name: "default.Table_1813e59368fb481cbaa4c146b45081ce" modify_external_catalogs: true
I20250109 20:53:10.535598 1158 tablet_bootstrap.cc:492] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.535708 1158 ts_tablet_manager.cc:1397] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
W20250109 20:53:10.535717 989 leader_election.cc:343] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.536250 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f4bdda16554d9d81de5ca5c5205f4c" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:10.536406 1158 raft_consensus.cc:357] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.536566 1158 raft_consensus.cc:383] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.536619 1158 raft_consensus.cc:738] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
W20250109 20:53:10.536679 990 leader_election.cc:343] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.536794 990 leader_election.cc:304] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.536839 1158 consensus_queue.cc:260] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.537146 2749 raft_consensus.cc:2743] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.537313 1158 ts_tablet_manager.cc:1428] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:10.538275 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d9affec4411a4ff4aeb64926df8c7e47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.538610 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9affec4411a4ff4aeb64926df8c7e47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.539647 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 82209a6b39bb48e791ff63a8167c9765 (table default.Table_06b47da1327e425989c115839e9aecbe [id=cced112727b0498f955953f2b2d29551]) successfully deleted
W20250109 20:53:10.539819 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet b1e6beb38a994f02a6c1d7357c2dbd63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1e6beb38a994f02a6c1d7357c2dbd63 already in progress: creating tablet
W20250109 20:53:10.539958 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet af77e079fe5642d790707406c5588420 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af77e079fe5642d790707406c5588420 already in progress: creating tablet
W20250109 20:53:10.540097 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5c8d8d771b56407d94fcb23d76796039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8d8d771b56407d94fcb23d76796039 already in progress: creating tablet
W20250109 20:53:10.540239 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 6daac072067b4071bf2280f05c47f14b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6daac072067b4071bf2280f05c47f14b already in progress: creating tablet
I20250109 20:53:10.540496 1176 tablet_bootstrap.cc:492] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.540943 1158 tablet_bootstrap.cc:492] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.541872 1176 tablet_bootstrap.cc:654] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.542614 1158 tablet_bootstrap.cc:654] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.544226 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cfef853be74f4125ae5717179a124807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.544564 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfef853be74f4125ae5717179a124807 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.546018 1157 ts_tablet_manager.cc:1918] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.546125 1157 log.cc:1198] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/72a6cf525cb44c428f7a39a25c4e62dc
I20250109 20:53:10.546409 1157 ts_tablet_manager.cc:1939] T 72a6cf525cb44c428f7a39a25c4e62dc P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:10.547276 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 72a6cf525cb44c428f7a39a25c4e62dc (table default.Table_5aceb6f20bb841acb677692231b8f670 [id=a662d40e49744d56842c52a82df4df13]) successfully deleted
I20250109 20:53:10.547919 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f59d59762b8b4f55b3123c4df5bfd12e" } modify_external_catalogs: true
I20250109 20:53:10.548053 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f59d59762b8b4f55b3123c4df5bfd12e" } modify_external_catalogs: true
I20250109 20:53:10.549115 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c308f00a936a4895b1d9d35b5df4b50a" } modify_external_catalogs: true
I20250109 20:53:10.549252 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c308f00a936a4895b1d9d35b5df4b50a" } modify_external_catalogs: true
I20250109 20:53:10.549686 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_38af1d150640411a90cf87f30d618cb5" } new_table_name: "default.Table_e8f3c2466e9646bb8e213633b84850ec" modify_external_catalogs: true
I20250109 20:53:10.550654 1166 ts_tablet_manager.cc:1918] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.550765 1166 log.cc:1198] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/faeec3dbcaff4de98e94357f20e7e8ad
I20250109 20:53:10.551682 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_e6ba223109264a98a2a80e20e982b0ba [id=f91ce8749a344785a06059e5e4fa97b6] alter complete (version 1)
W20250109 20:53:10.552537 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6e895173440e48faafe5e9bdc37051b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e895173440e48faafe5e9bdc37051b5 already in progress: creating tablet
W20250109 20:53:10.553033 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet de57e94d00d748dc8b02d60f2f1fb27a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de57e94d00d748dc8b02d60f2f1fb27a already in progress: creating tablet
I20250109 20:53:10.553486 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f953f43877c4a738ffd8255afbf8275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:10.553668 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 417aab64aaef487a83c1e0ff6eec9ffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417aab64aaef487a83c1e0ff6eec9ffc already in progress: creating tablet
I20250109 20:53:10.553714 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d57a8bbb3da84088a98c816102e4e9df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.553891 1160 tablet_replica.cc:331] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:10.554066 1160 raft_consensus.cc:2238] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.554215 1160 raft_consensus.cc:2267] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.554873 1160 ts_tablet_manager.cc:1905] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:10.555250 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f12b13cead8942e2a4e874dde17a0802 consensus state has no leader
I20250109 20:53:10.551017 1166 ts_tablet_manager.cc:1939] T faeec3dbcaff4de98e94357f20e7e8ad P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.556463 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet faeec3dbcaff4de98e94357f20e7e8ad (table default.Table_472ddec2dea04b1d83cb147944a488c9 [id=2da6a2c99a6e40d687307f107bef5264]) successfully deleted
W20250109 20:53:10.558423 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 070477648d3e49d0af77fbf594d8ef77 consensus state has no leader
I20250109 20:53:10.560540 1160 ts_tablet_manager.cc:1918] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.560626 1160 log.cc:1198] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/5f953f43877c4a738ffd8255afbf8275
W20250109 20:53:10.560705 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet af77e079fe5642d790707406c5588420 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af77e079fe5642d790707406c5588420 already in progress: creating tablet
W20250109 20:53:10.561465 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet b1e6beb38a994f02a6c1d7357c2dbd63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1e6beb38a994f02a6c1d7357c2dbd63 already in progress: creating tablet
W20250109 20:53:10.561887 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 30e9398f839f4604819e82794af94351 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30e9398f839f4604819e82794af94351 already in progress: creating tablet
I20250109 20:53:10.564018 1160 ts_tablet_manager.cc:1939] T 5f953f43877c4a738ffd8255afbf8275 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:10.580716 1176 tablet_bootstrap.cc:492] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.580893 1176 ts_tablet_manager.cc:1397] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250109 20:53:10.581256 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 5f953f43877c4a738ffd8255afbf8275 (table default.Table_3254b432932b4d9bb83cb8a2f26e5514 [id=fa8fa63dbd44442b81a3b9640d42680e]) successfully deleted
I20250109 20:53:10.581321 1160 tablet_replica.cc:331] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:10.581503 1160 raft_consensus.cc:2238] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.581544 2695 raft_consensus.cc:491] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.581672 2695 raft_consensus.cc:513] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.582212 2695 leader_election.cc:290] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.582682 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.583173 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
W20250109 20:53:10.583252 712 leader_election.cc:343] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.583329 1051 raft_consensus.cc:2463] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:10.583451 1160 raft_consensus.cc:2267] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.583665 713 leader_election.cc:304] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.584024 2695 raft_consensus.cc:2798] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:10.584086 2695 raft_consensus.cc:491] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:10.584165 2695 raft_consensus.cc:3054] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.587006 2695 raft_consensus.cc:513] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.587801 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:10.588228 712 leader_election.cc:343] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.590454 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:10.590610 1051 raft_consensus.cc:3054] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.594233 1051 raft_consensus.cc:2463] T ee208bad76734a8fa1ada438bdbbe48c P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:10.594566 713 leader_election.cc:304] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.594766 2692 raft_consensus.cc:2798] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:10.594841 2692 raft_consensus.cc:695] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:10.595064 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1db09bd8ba8c4c3fb3c525c18eb0e989"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:10.595137 2692 consensus_queue.cc:237] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:10.595502 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1db09bd8ba8c4c3fb3c525c18eb0e989 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.596305 1160 ts_tablet_manager.cc:1905] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.596122 451 catalog_manager.cc:5526] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:10.596465 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 2769e5906c604bd5b14df79db765ec61 (table default.Table_f59d59762b8b4f55b3123c4df5bfd12e [id=3c9451140e48432f8591841d4dfd8751]). Expected version 5 got 0
I20250109 20:53:10.597149 2695 leader_election.cc:290] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.599406 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 1132c08b23ea4849a78b4809cbe44360
I20250109 20:53:10.598930 1176 raft_consensus.cc:357] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.599619 1176 raft_consensus.cc:383] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.599813 1176 raft_consensus.cc:738] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.600098 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 1132c08b23ea4849a78b4809cbe44360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.600137 1176 consensus_queue.cc:260] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.601040 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_680621376a2449aeb32ebcd741a33b5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.601423 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_680621376a2449aeb32ebcd741a33b5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.603335 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 1132c08b23ea4849a78b4809cbe44360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.604959 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 1132c08b23ea4849a78b4809cbe44360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:10.605226 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 533ac2bcf30d4ff091c5c0d5e418fc70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533ac2bcf30d4ff091c5c0d5e418fc70 already in progress: creating tablet
I20250109 20:53:10.605511 1157 tablet_replica.cc:331] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:10.605692 1157 raft_consensus.cc:2238] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.605827 1157 raft_consensus.cc:2267] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.606375 1157 ts_tablet_manager.cc:1905] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.604599 1165 raft_consensus.cc:357] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.608863 1165 raft_consensus.cc:383] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.608939 1165 raft_consensus.cc:738] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.609261 1165 consensus_queue.cc:260] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.611115 455 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 9629fe30752f49478afad9b93dc235eb (table default.Table_df4e58889f9d4ef6b81876c48ecf223e [id=fcb76535fbbe44e29ef627a3ae562ce2]). Expected version 5 got 0
I20250109 20:53:10.612282 1158 tablet_bootstrap.cc:492] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.612406 1158 ts_tablet_manager.cc:1397] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.003s
I20250109 20:53:10.613075 2695 raft_consensus.cc:491] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.613128 1158 raft_consensus.cc:357] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.613190 2695 raft_consensus.cc:513] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.613273 1158 raft_consensus.cc:383] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.613314 1158 raft_consensus.cc:738] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.613464 1158 consensus_queue.cc:260] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.613919 1158 ts_tablet_manager.cc:1428] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:10.613997 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6461f153a5994412ba05c3b687963fd1" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.614173 1158 tablet_bootstrap.cc:492] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:10.614416 712 leader_election.cc:343] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.615002 2695 leader_election.cc:290] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:10.615608 713 leader_election.cc:343] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 6461f153a5994412ba05c3b687963fd1
I20250109 20:53:10.615706 713 leader_election.cc:304] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:10.616370 1158 tablet_bootstrap.cc:654] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.616371 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e86358bf98744df4a2432305fcffde2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.616809 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e86358bf98744df4a2432305fcffde2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.616956 2695 raft_consensus.cc:2743] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.619376 898 tablet_service.cc:1467] Processing CreateTablet for tablet 3e45125e7ca14225b67c0cf92f76b312 (DEFAULT_TABLE table=default.Table_2883eb1cae054dc4b3073c4944e5355f [id=41d7973a58d44adc97780e728aab4190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.619694 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45125e7ca14225b67c0cf92f76b312. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.620074 1176 ts_tablet_manager.cc:1428] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20250109 20:53:10.620383 1176 tablet_bootstrap.cc:492] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.627615 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3e45125e7ca14225b67c0cf92f76b312 (DEFAULT_TABLE table=default.Table_2883eb1cae054dc4b3073c4944e5355f [id=41d7973a58d44adc97780e728aab4190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.627935 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45125e7ca14225b67c0cf92f76b312. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.632791 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2769e5906c604bd5b14df79db765ec61
I20250109 20:53:10.633180 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a7aa8b2c8e744aa3a1ac70fef238e65b
I20250109 20:53:10.634034 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 9719bbee9dbc4934992b47bf7b9bd249
I20250109 20:53:10.634963 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 9719bbee9dbc4934992b47bf7b9bd249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.635615 1176 tablet_bootstrap.cc:654] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.637208 447 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 9629fe30752f49478afad9b93dc235eb (table default.Table_df4e58889f9d4ef6b81876c48ecf223e [id=fcb76535fbbe44e29ef627a3ae562ce2]). Expected version 5 got 0
I20250109 20:53:10.638267 742 tablet_service.cc:1514] Processing DeleteTablet for tablet a7aa8b2c8e744aa3a1ac70fef238e65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.638422 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 9719bbee9dbc4934992b47bf7b9bd249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.639474 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2769e5906c604bd5b14df79db765ec61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.639961 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet a7aa8b2c8e744aa3a1ac70fef238e65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.640115 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 9719bbee9dbc4934992b47bf7b9bd249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.640610 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2769e5906c604bd5b14df79db765ec61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.641115 1165 ts_tablet_manager.cc:1428] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20250109 20:53:10.641409 1165 tablet_bootstrap.cc:492] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:10.642653 1165 tablet_bootstrap.cc:654] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:10.643613 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5c8d8d771b56407d94fcb23d76796039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8d8d771b56407d94fcb23d76796039 already in progress: creating tablet
I20250109 20:53:10.643698 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 2769e5906c604bd5b14df79db765ec61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.643886 1166 tablet_replica.cc:331] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:10.644061 1166 raft_consensus.cc:2238] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.644201 1166 raft_consensus.cc:2267] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.646071 1160 ts_tablet_manager.cc:1918] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.646250 1160 log.cc:1198] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/d57a8bbb3da84088a98c816102e4e9df
I20250109 20:53:10.646603 1160 ts_tablet_manager.cc:1939] T d57a8bbb3da84088a98c816102e4e9df P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:10.647205 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cede2a1882946a181b18d66f12c5114 with delete_type TABLET_DATA_DELETED (replaced by tablet 1343e8e822684fea8a13d8df9ec33779) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.647725 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet d57a8bbb3da84088a98c816102e4e9df (table default.Table_634a6ecf6a7b4731ae4ac6e03c5375d1 [id=0bf46974692d47c38befe7eca4f23cc9]) successfully deleted
I20250109 20:53:10.649195 1166 ts_tablet_manager.cc:1905] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.650981 898 tablet_service.cc:1514] Processing DeleteTablet for tablet a7aa8b2c8e744aa3a1ac70fef238e65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.651894 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 3e45125e7ca14225b67c0cf92f76b312 (DEFAULT_TABLE table=default.Table_2883eb1cae054dc4b3073c4944e5355f [id=41d7973a58d44adc97780e728aab4190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.652235 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45125e7ca14225b67c0cf92f76b312. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.658738 898 tablet_service.cc:1467] Processing CreateTablet for tablet c541dff536224aee8ce44caba4666a6b (DEFAULT_TABLE table=default.Table_d1cde7e6cd224450a38f47398b23e38c [id=12ff99447b7a46bc995026218a1191f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.659344 2695 raft_consensus.cc:491] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.659543 2695 raft_consensus.cc:513] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.659682 749 tablet_service.cc:1467] Processing CreateTablet for tablet c541dff536224aee8ce44caba4666a6b (DEFAULT_TABLE table=default.Table_d1cde7e6cd224450a38f47398b23e38c [id=12ff99447b7a46bc995026218a1191f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.660048 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c541dff536224aee8ce44caba4666a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.660543 1157 ts_tablet_manager.cc:1918] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.661157 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc54e97e91c54afea098624bd85fcc6e" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:10.661557 1051 raft_consensus.cc:2463] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:10.661924 1165 tablet_bootstrap.cc:492] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:10.662055 1165 ts_tablet_manager.cc:1397] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20250109 20:53:10.662914 1165 raft_consensus.cc:357] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.663110 1165 raft_consensus.cc:383] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.663179 1165 raft_consensus.cc:738] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.663388 1165 consensus_queue.cc:260] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.663892 1165 ts_tablet_manager.cc:1428] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:10.664431 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c541dff536224aee8ce44caba4666a6b (DEFAULT_TABLE table=default.Table_d1cde7e6cd224450a38f47398b23e38c [id=12ff99447b7a46bc995026218a1191f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.664790 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c541dff536224aee8ce44caba4666a6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:10.668835 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 50306.8us
I20250109 20:53:10.670286 881 tablet_service.cc:1467] Processing CreateTablet for tablet d3139e0aec4a4300b10750e487ee78bf (DEFAULT_TABLE table=default.Table_a56fbc8f36a2420a992dea0a8126723a [id=25d7f3d36299434d9a6c058d2529d4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.672904 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc54e97e91c54afea098624bd85fcc6e" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.673018 912 raft_consensus.cc:3054] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.673230 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d3139e0aec4a4300b10750e487ee78bf (DEFAULT_TABLE table=default.Table_a56fbc8f36a2420a992dea0a8126723a [id=25d7f3d36299434d9a6c058d2529d4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.673533 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3139e0aec4a4300b10750e487ee78bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.683383 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3139e0aec4a4300b10750e487ee78bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.689249 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c541dff536224aee8ce44caba4666a6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:10.691423 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 1132c08b23ea4849a78b4809cbe44360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1132c08b23ea4849a78b4809cbe44360 already in progress: creating tablet
I20250109 20:53:10.691896 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e36aa38ac6b4bf191f0652d3e3b0691" } modify_external_catalogs: true
I20250109 20:53:10.692034 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2e36aa38ac6b4bf191f0652d3e3b0691" } modify_external_catalogs: true
I20250109 20:53:10.693107 1160 tablet_replica.cc:331] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:10.693279 1160 raft_consensus.cc:2238] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.694274 1176 tablet_bootstrap.cc:492] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
W20250109 20:53:10.694380 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 9719bbee9dbc4934992b47bf7b9bd249 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9719bbee9dbc4934992b47bf7b9bd249 already in progress: creating tablet
I20250109 20:53:10.694408 1176 ts_tablet_manager.cc:1397] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
W20250109 20:53:10.694552 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2769e5906c604bd5b14df79db765ec61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2769e5906c604bd5b14df79db765ec61 already in progress: creating tablet
I20250109 20:53:10.695055 1176 raft_consensus.cc:357] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.693406 1160 raft_consensus.cc:2267] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.697486 1160 ts_tablet_manager.cc:1905] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.698976 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8594ac97908c49dba72c916b11b44850"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.699445 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8594ac97908c49dba72c916b11b44850 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.702210 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d0a5a92a65dc4547881bf914b88c9b84
I20250109 20:53:10.703217 1165 tablet_bootstrap.cc:492] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:10.703510 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d0a5a92a65dc4547881bf914b88c9b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.704509 1165 tablet_bootstrap.cc:654] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.705469 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7eaae83ca44c4bdcae00fed3b9e2a6d8" } new_table_name: "default.Table_1968159c258f416a8340336e01f662a3" modify_external_catalogs: true
I20250109 20:53:10.662045 713 leader_election.cc:304] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:10.706656 1157 log.cc:1198] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/deedac68322e460b8d62adbe849de3f2
I20250109 20:53:10.662201 2695 leader_election.cc:290] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:10.706857 712 leader_election.cc:343] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.671105 746 tablet_service.cc:1467] Processing CreateTablet for tablet d3139e0aec4a4300b10750e487ee78bf (DEFAULT_TABLE table=default.Table_a56fbc8f36a2420a992dea0a8126723a [id=25d7f3d36299434d9a6c058d2529d4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.706995 1157 ts_tablet_manager.cc:1939] T deedac68322e460b8d62adbe849de3f2 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:10.707293 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3139e0aec4a4300b10750e487ee78bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.707801 1176 raft_consensus.cc:383] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.707868 1176 raft_consensus.cc:738] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.708076 1176 consensus_queue.cc:260] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.703748 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet d0a5a92a65dc4547881bf914b88c9b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.708581 1176 ts_tablet_manager.cc:1428] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250109 20:53:10.708932 1176 tablet_bootstrap.cc:492] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.710441 1176 tablet_bootstrap.cc:654] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.712405 881 tablet_service.cc:1467] Processing CreateTablet for tablet bcbcc1b88f2740d9ae7bc36b4684b552 (DEFAULT_TABLE table=default.Table_eaba7b7ff5d44921a1ce8461ba178fbc [id=a0a77d2aa39d47d4a3aefd3c2ea2e34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.712738 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcbcc1b88f2740d9ae7bc36b4684b552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.713662 1166 ts_tablet_manager.cc:1918] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.713753 1166 log.cc:1198] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/d4c0f80ec2094da0893fb0a92223b1cd
W20250109 20:53:10.713903 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 51879.2us
I20250109 20:53:10.713941 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d0a5a92a65dc4547881bf914b88c9b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.714015 1166 ts_tablet_manager.cc:1939] T d4c0f80ec2094da0893fb0a92223b1cd P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.714243 2754 raft_consensus.cc:2798] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:10.714344 2754 raft_consensus.cc:491] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:10.714417 2754 raft_consensus.cc:3054] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.715313 2692 raft_consensus.cc:491] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.715440 2692 raft_consensus.cc:513] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.716130 2692 leader_election.cc:290] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.717216 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet d4c0f80ec2094da0893fb0a92223b1cd (table default.Table_dcb1259a70b64057b0090a78234d47a6 [id=80c904806c7444329fbca174ae613fc6]) successfully deleted
W20250109 20:53:10.718104 2792 log.cc:926] Time spent T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20250109 20:53:10.718607 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 45202.3us
I20250109 20:53:10.718897 1165 tablet_bootstrap.cc:492] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:10.719020 1165 ts_tablet_manager.cc:1397] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20250109 20:53:10.720060 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b4c038573a4afa879ecbcba75c35a7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:10.723011 712 leader_election.cc:343] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.723377 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_bb47b9b8617f4706aeb895a16ded22ee" } new_table_name: "default.Table_c9b68cce9a8544a38a418f69637719b6" modify_external_catalogs: true
I20250109 20:53:10.725062 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1ce8dba031c24ab2bd13cbaa5d99c9fd" } new_table_name: "default.Table_b5d7a1ede01b40f88002abdc64963e96" modify_external_catalogs: true
W20250109 20:53:10.729180 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 1132c08b23ea4849a78b4809cbe44360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1132c08b23ea4849a78b4809cbe44360 already in progress: creating tablet
W20250109 20:53:10.730557 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a7aa8b2c8e744aa3a1ac70fef238e65b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7aa8b2c8e744aa3a1ac70fef238e65b already in progress: creating tablet
I20250109 20:53:10.732430 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1d1de589c7d742b2a63f6216b5655247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.732910 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d1de589c7d742b2a63f6216b5655247 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:10.733129 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 9719bbee9dbc4934992b47bf7b9bd249 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9719bbee9dbc4934992b47bf7b9bd249 already in progress: creating tablet
I20250109 20:53:10.733431 1165 raft_consensus.cc:357] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.733609 1165 raft_consensus.cc:383] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.733655 1165 raft_consensus.cc:738] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.734357 1166 tablet_replica.cc:331] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:10.734546 1166 raft_consensus.cc:2238] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.736160 1166 raft_consensus.cc:2267] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.736728 1166 ts_tablet_manager.cc:1905] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.738025 1165 consensus_queue.cc:260] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.738570 1165 ts_tablet_manager.cc:1428] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20250109 20:53:10.738843 1165 tablet_bootstrap.cc:492] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:10.739248 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet deedac68322e460b8d62adbe849de3f2 (table default.Table_17aab5c8aa9a4999b2eebd66dcae9f46 [id=3bdc7c0be0d14815a92667bde1011e67]) successfully deleted
W20250109 20:53:10.741122 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e5820a9fe844227aba8ed848605dcc1 consensus state has no leader
W20250109 20:53:10.741566 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 355a7dc898bf4b4c9bb49e63cf999017 consensus state has no leader
I20250109 20:53:10.741861 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f18f244a344272bba2bc085686b61d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:10.743654 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2420b285905342b3820f9549c8f0c4ad consensus state has no leader
I20250109 20:53:10.744222 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 64d462969643467c97222cac8adc210d from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:10.745165 1020 tablet_service.cc:1467] Processing CreateTablet for tablet bcbcc1b88f2740d9ae7bc36b4684b552 (DEFAULT_TABLE table=default.Table_eaba7b7ff5d44921a1ce8461ba178fbc [id=a0a77d2aa39d47d4a3aefd3c2ea2e34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.745537 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcbcc1b88f2740d9ae7bc36b4684b552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.746135 1157 tablet_replica.cc:331] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:10.747262 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_48c282098fff4f29af36cc85c62ec34a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.747704 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48c282098fff4f29af36cc85c62ec34a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.747694 1157 raft_consensus.cc:2238] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:10.748014 1157 pending_rounds.cc:62] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Trying to abort 1 pending ops.
I20250109 20:53:10.748103 1157 pending_rounds.cc:69] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112523736000667648 op_type: NO_OP noop_request { }
I20250109 20:53:10.749326 1157 raft_consensus.cc:2883] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250109 20:53:10.749568 1176 tablet_bootstrap.cc:492] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.749691 1176 ts_tablet_manager.cc:1397] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250109 20:53:10.750162 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1d3219f0687d4cb595ce0dae72cfb41a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:10.750367 1176 raft_consensus.cc:357] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:10.750560 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d3219f0687d4cb595ce0dae72cfb41a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.750566 1176 raft_consensus.cc:383] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.750630 1176 raft_consensus.cc:738] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.750830 1176 consensus_queue.cc:260] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.751577 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_00c020c7924a4724baf7a9d36a9971b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.751891 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00c020c7924a4724baf7a9d36a9971b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.752799 1157 raft_consensus.cc:2267] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.753069 1160 ts_tablet_manager.cc:1918] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.756119 1160 log.cc:1198] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/0cede2a1882946a181b18d66f12c5114
I20250109 20:53:10.756847 895 tablet_service.cc:1467] Processing CreateTablet for tablet ab5bfc387bc24feeb73bf5ce44af27d0 (DEFAULT_TABLE table=default.Table_b652be7142be44bab45b5ad83a328a06 [id=9ec1ec336aca4bdfaa44fbc863b330ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.757510 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab5bfc387bc24feeb73bf5ce44af27d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.759060 1165 tablet_bootstrap.cc:654] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.760911 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_0861a5b375f1434abfc4c91f0ae5dd2c" } new_table_name: "default.Table_b7f7f29cdf9f4b1a97f339853d26a6b2" modify_external_catalogs: true
I20250109 20:53:10.762180 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3a973775fce94f7493d3ca21e4698bf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.762544 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a973775fce94f7493d3ca21e4698bf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.763402 1176 ts_tablet_manager.cc:1428] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250109 20:53:10.763670 1176 tablet_bootstrap.cc:492] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.759264 1160 ts_tablet_manager.cc:1939] T 0cede2a1882946a181b18d66f12c5114 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:10.765244 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8252859a8fcd44b0984a8c054e3c0416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.765599 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8252859a8fcd44b0984a8c054e3c0416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.765765 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 0cede2a1882946a181b18d66f12c5114 (table default.Table_69f93e68f71f4bc589abb0edbc340e6c [id=d53db1b02c784a4ebc747540a200d237]) successfully deleted
W20250109 20:53:10.766510 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a7aa8b2c8e744aa3a1ac70fef238e65b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7aa8b2c8e744aa3a1ac70fef238e65b already in progress: creating tablet
I20250109 20:53:10.763484 915 raft_consensus.cc:3054] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.766876 448 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 049321d28ab241138247dcefa0411f54 (table default.Table_800bb021710b4f01a0769f08ca4f5fb1 [id=73d86ca9226d4a84aacc05835ada9e03]). Expected version 1 got 0
W20250109 20:53:10.772552 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d0a5a92a65dc4547881bf914b88c9b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0a5a92a65dc4547881bf914b88c9b84 already in progress: creating tablet
I20250109 20:53:10.773559 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_8ee14609a1444a7f853f33a1c27476cd" } new_table_name: "default.Table_1e5d4f2960e444ecbc9a20662e2bf4b4" modify_external_catalogs: true
W20250109 20:53:10.775183 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a85def613874f98a0c9566c5b0a35a0 consensus state has no leader
I20250109 20:53:10.778235 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 049321d28ab241138247dcefa0411f54 (table default.Table_800bb021710b4f01a0769f08ca4f5fb1 [id=73d86ca9226d4a84aacc05835ada9e03]). Expected version 1 got 0
I20250109 20:53:10.778976 1157 ts_tablet_manager.cc:1905] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:10.785238 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd7f542fe5c94de99c369e291d39304d consensus state has no leader
I20250109 20:53:10.789752 1176 tablet_bootstrap.cc:654] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.790083 1031 tablet_service.cc:1467] Processing CreateTablet for tablet ab5bfc387bc24feeb73bf5ce44af27d0 (DEFAULT_TABLE table=default.Table_b652be7142be44bab45b5ad83a328a06 [id=9ec1ec336aca4bdfaa44fbc863b330ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.790490 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab5bfc387bc24feeb73bf5ce44af27d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.792274 898 tablet_service.cc:1467] Processing CreateTablet for tablet 34e71dfad34b49438e9795a95413a520 (DEFAULT_TABLE table=default.Table_4800f1d4b4d54e4b9abef2016f1684c9 [id=24e228dba65c492d9bc169dba9d66207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.793738 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e71dfad34b49438e9795a95413a520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.794689 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 34e71dfad34b49438e9795a95413a520 (DEFAULT_TABLE table=default.Table_4800f1d4b4d54e4b9abef2016f1684c9 [id=24e228dba65c492d9bc169dba9d66207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.794989 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e71dfad34b49438e9795a95413a520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.795984 1165 tablet_bootstrap.cc:492] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:10.796120 1165 ts_tablet_manager.cc:1397] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.004s
I20250109 20:53:10.796159 881 tablet_service.cc:1467] Processing CreateTablet for tablet 60f09b5be5ea42db8f71ea9a44be8b1b (DEFAULT_TABLE table=default.Table_df7d6213072043f5b9348690ddce3ceb [id=412ef02449c048c090feff99817ea6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.796465 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f09b5be5ea42db8f71ea9a44be8b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.796859 1165 raft_consensus.cc:357] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.797055 1165 raft_consensus.cc:383] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.797134 1165 raft_consensus.cc:738] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.797336 1165 consensus_queue.cc:260] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.797775 1165 ts_tablet_manager.cc:1428] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:10.798751 1176 tablet_bootstrap.cc:492] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.798858 1176 ts_tablet_manager.cc:1397] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.001s
I20250109 20:53:10.800110 1176 raft_consensus.cc:357] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.800271 1176 raft_consensus.cc:383] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.800309 1176 raft_consensus.cc:738] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.800487 1176 consensus_queue.cc:260] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.801064 2788 tablet.cc:1721] T 5ce0a941f0c14501a947a5c7e2a844e8 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:10.801393 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 60f09b5be5ea42db8f71ea9a44be8b1b (DEFAULT_TABLE table=default.Table_df7d6213072043f5b9348690ddce3ceb [id=412ef02449c048c090feff99817ea6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.801674 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f09b5be5ea42db8f71ea9a44be8b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.806151 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f53efacb818d49bea258bd79021056d3" } modify_external_catalogs: true
I20250109 20:53:10.806330 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f53efacb818d49bea258bd79021056d3" } modify_external_catalogs: true
I20250109 20:53:10.810957 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bd42ddc2011e4030b0063d864e6972db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.811429 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd42ddc2011e4030b0063d864e6972db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.814759 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 70b48900f57f47128e31b27275b1489c
I20250109 20:53:10.815816 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b48900f57f47128e31b27275b1489c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.816494 1160 tablet_replica.cc:331] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:10.816787 1160 raft_consensus.cc:2238] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.817027 1160 raft_consensus.cc:2267] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.817391 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b48900f57f47128e31b27275b1489c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.817606 1160 ts_tablet_manager.cc:1905] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:10.819106 895 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d74fde6bf4e1792a18bd904d4b0b7 (DEFAULT_TABLE table=default.Table_9a175bd72fd749ac80c63aae2b1567a5 [id=2625906405cf4fdcbb83efa610789aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.819422 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5d74fde6bf4e1792a18bd904d4b0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.826843 746 tablet_service.cc:1467] Processing CreateTablet for tablet bcbcc1b88f2740d9ae7bc36b4684b552 (DEFAULT_TABLE table=default.Table_eaba7b7ff5d44921a1ce8461ba178fbc [id=a0a77d2aa39d47d4a3aefd3c2ea2e34b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.827247 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcbcc1b88f2740d9ae7bc36b4684b552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.827441 1166 ts_tablet_manager.cc:1918] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.827576 1166 log.cc:1198] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2769e5906c604bd5b14df79db765ec61
I20250109 20:53:10.827872 1166 ts_tablet_manager.cc:1939] T 2769e5906c604bd5b14df79db765ec61 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.828174 742 tablet_service.cc:1467] Processing CreateTablet for tablet ab5bfc387bc24feeb73bf5ce44af27d0 (DEFAULT_TABLE table=default.Table_b652be7142be44bab45b5ad83a328a06 [id=9ec1ec336aca4bdfaa44fbc863b330ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.828539 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab5bfc387bc24feeb73bf5ce44af27d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.832341 752 tablet_service.cc:1467] Processing CreateTablet for tablet 34e71dfad34b49438e9795a95413a520 (DEFAULT_TABLE table=default.Table_4800f1d4b4d54e4b9abef2016f1684c9 [id=24e228dba65c492d9bc169dba9d66207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.832706 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e71dfad34b49438e9795a95413a520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.833478 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b48900f57f47128e31b27275b1489c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.833813 740 tablet_service.cc:1467] Processing CreateTablet for tablet 60f09b5be5ea42db8f71ea9a44be8b1b (DEFAULT_TABLE table=default.Table_df7d6213072043f5b9348690ddce3ceb [id=412ef02449c048c090feff99817ea6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.834139 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f09b5be5ea42db8f71ea9a44be8b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.834556 2749 raft_consensus.cc:491] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.834677 2749 raft_consensus.cc:513] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.835271 2749 leader_election.cc:290] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:10.835992 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d455aef15046e793f97f0ba2c994ed" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:10.836465 990 leader_election.cc:343] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.838578 2692 raft_consensus.cc:491] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.838706 2692 raft_consensus.cc:513] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.838945 1176 ts_tablet_manager.cc:1428] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20250109 20:53:10.839188 1176 tablet_bootstrap.cc:492] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.839342 2692 leader_election.cc:290] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.839594 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d455aef15046e793f97f0ba2c994ed" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:10.840097 989 leader_election.cc:343] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.840199 989 leader_election.cc:304] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.840551 2749 raft_consensus.cc:2743] T c5d455aef15046e793f97f0ba2c994ed P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.840620 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "face19003d0042669e3f22553f9fe19a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:10.841248 1166 tablet_replica.cc:331] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:53:10.841233 712 leader_election.cc:343] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.841420 1166 raft_consensus.cc:2238] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:10.841547 1166 raft_consensus.cc:2267] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:10.841601 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2769e5906c604bd5b14df79db765ec61 (table default.Table_f59d59762b8b4f55b3123c4df5bfd12e [id=3c9451140e48432f8591841d4dfd8751]) successfully deleted
W20250109 20:53:10.841863 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d0a5a92a65dc4547881bf914b88c9b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0a5a92a65dc4547881bf914b88c9b84 already in progress: creating tablet
W20250109 20:53:10.842043 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 6daac072067b4071bf2280f05c47f14b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6daac072067b4071bf2280f05c47f14b already in progress: creating tablet
W20250109 20:53:10.842204 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet af77e079fe5642d790707406c5588420 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af77e079fe5642d790707406c5588420 already in progress: creating tablet
I20250109 20:53:10.842206 1166 ts_tablet_manager.cc:1905] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:10.842344 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5c8d8d771b56407d94fcb23d76796039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8d8d771b56407d94fcb23d76796039 already in progress: creating tablet
W20250109 20:53:10.842489 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet b1e6beb38a994f02a6c1d7357c2dbd63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1e6beb38a994f02a6c1d7357c2dbd63 already in progress: creating tablet
W20250109 20:53:10.842650 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 1132c08b23ea4849a78b4809cbe44360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1132c08b23ea4849a78b4809cbe44360 already in progress: creating tablet
W20250109 20:53:10.842792 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet a7aa8b2c8e744aa3a1ac70fef238e65b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7aa8b2c8e744aa3a1ac70fef238e65b already in progress: creating tablet
W20250109 20:53:10.842926 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 9719bbee9dbc4934992b47bf7b9bd249 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9719bbee9dbc4934992b47bf7b9bd249 already in progress: creating tablet
I20250109 20:53:10.844563 1165 tablet_bootstrap.cc:492] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:53:10.845822 2144 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:55611 (request call id 137) took 950 ms (client timeout 1000 ms). Trace:
W20250109 20:53:10.846001 2144 rpcz_store.cc:269] 0109 20:53:09.894795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:09.901267 (+ 6472us) service_pool.cc:225] Handling call
0109 20:53:10.845727 (+944460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250109 20:53:10.847399 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_567d3232cbbd4fc0a087d039f51d5144" } modify_external_catalogs: true
I20250109 20:53:10.847555 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_567d3232cbbd4fc0a087d039f51d5144" } modify_external_catalogs: true
I20250109 20:53:10.850235 1165 tablet_bootstrap.cc:654] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.851697 1160 ts_tablet_manager.cc:1918] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:10.851794 1160 log.cc:1198] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/70b48900f57f47128e31b27275b1489c
I20250109 20:53:10.852113 1160 ts_tablet_manager.cc:1939] T 70b48900f57f47128e31b27275b1489c P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:10.854983 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 189cd8079a9d4ca6b1703f4361d5ec5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.856576 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_83001a14d61f44ec8e047a6fe2dccd69" } modify_external_catalogs: true
I20250109 20:53:10.857098 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_83001a14d61f44ec8e047a6fe2dccd69" } modify_external_catalogs: true
I20250109 20:53:10.858198 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a017309fe2384e848ad25b4288eb890b" } modify_external_catalogs: true
I20250109 20:53:10.861207 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a017309fe2384e848ad25b4288eb890b" } modify_external_catalogs: true
I20250109 20:53:10.862902 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e53966daf1ed4a67801d9ab239e17d8e
W20250109 20:53:10.864534 2203 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:36972 (request call id 0) took 994 ms (client timeout 1000 ms). Trace:
I20250109 20:53:10.865465 1176 tablet_bootstrap.cc:654] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.866312 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e53966daf1ed4a67801d9ab239e17d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:10.867406 2203 rpcz_store.cc:269] 0109 20:53:09.869907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:09.869973 (+ 66us) service_pool.cc:225] Handling call
0109 20:53:10.864501 (+994528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250109 20:53:10.868541 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2298004317b14932bf80f39568ad73da
I20250109 20:53:10.868625 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 63a665fcf6ba43f59de8b945c5534d8e
I20250109 20:53:10.869343 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a665fcf6ba43f59de8b945c5534d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.869558 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 2298004317b14932bf80f39568ad73da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.869735 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 2298004317b14932bf80f39568ad73da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.869928 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2298004317b14932bf80f39568ad73da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.870599 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a665fcf6ba43f59de8b945c5534d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.871174 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet e53966daf1ed4a67801d9ab239e17d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.871369 895 tablet_service.cc:1514] Processing DeleteTablet for tablet e53966daf1ed4a67801d9ab239e17d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.871186 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a665fcf6ba43f59de8b945c5534d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.871719 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_025053d76dd84c2a9a5d320fa3a085d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.872103 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_025053d76dd84c2a9a5d320fa3a085d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.878731 1157 ts_tablet_manager.cc:1918] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250109 20:53:10.879901 1176 tablet_bootstrap.cc:492] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.880024 1176 ts_tablet_manager.cc:1397] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20250109 20:53:10.880648 1176 raft_consensus.cc:357] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.880923 1176 raft_consensus.cc:383] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.881071 1176 raft_consensus.cc:738] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.881292 1176 consensus_queue.cc:260] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.884490 2633 raft_consensus.cc:491] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.884637 2633 raft_consensus.cc:513] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.885339 2633 leader_election.cc:290] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:10.886327 854 leader_election.cc:343] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 0ec1d30b67674facb8b92a2c1a408073
W20250109 20:53:10.886541 854 leader_election.cc:343] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 0ec1d30b67674facb8b92a2c1a408073
I20250109 20:53:10.886636 854 leader_election.cc:304] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:10.887295 2633 raft_consensus.cc:2743] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.888821 2749 raft_consensus.cc:491] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.888945 2749 raft_consensus.cc:513] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.889530 2749 leader_election.cc:290] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:10.890107 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f407b76780974cf7b3635fa1bf1585ec" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:10.890590 989 leader_election.cc:343] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.891263 2754 raft_consensus.cc:513] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.891538 1176 ts_tablet_manager.cc:1428] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250109 20:53:10.891782 1176 tablet_bootstrap.cc:492] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.891813 2754 leader_election.cc:290] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.892158 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f407b76780974cf7b3635fa1bf1585ec" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:10.892598 990 leader_election.cc:343] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.892685 990 leader_election.cc:304] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:10.893113 2749 raft_consensus.cc:2743] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.894433 742 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d74fde6bf4e1792a18bd904d4b0b7 (DEFAULT_TABLE table=default.Table_9a175bd72fd749ac80c63aae2b1567a5 [id=2625906405cf4fdcbb83efa610789aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.894627 1176 tablet_bootstrap.cc:654] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.894644 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc54e97e91c54afea098624bd85fcc6e" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:10.895499 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet feb9b0f722eb44b0acd5cb3bbb32c527 with delete_type TABLET_DATA_DELETED (replaced by tablet d4b939cdea1c4719b9b3918aabbdfc0e) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.897509 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc54e97e91c54afea098624bd85fcc6e" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:10.897661 1050 raft_consensus.cc:3054] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:10.899745 1050 raft_consensus.cc:2463] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:10.899935 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b4c038573a4afa879ecbcba75c35a7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:10.900080 1051 raft_consensus.cc:2463] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:10.900198 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5d74fde6bf4e1792a18bd904d4b0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.901543 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_42e8514aea2f48edb963804d6577aa3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.902042 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42e8514aea2f48edb963804d6577aa3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:10.895059 712 leader_election.cc:343] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:10.903554 713 leader_election.cc:343] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: face19003d0042669e3f22553f9fe19a
I20250109 20:53:10.903638 713 leader_election.cc:304] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:10.904206 2695 raft_consensus.cc:2743] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:10.896639 2754 raft_consensus.cc:491] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:10.904567 2754 raft_consensus.cc:513] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.905057 2754 leader_election.cc:290] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:10.905153 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1f571e8b256c4b74bd690e2d61870d9f" } modify_external_catalogs: true
I20250109 20:53:10.905313 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_1f571e8b256c4b74bd690e2d61870d9f" } modify_external_catalogs: true
I20250109 20:53:10.907681 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2efcc2a6ec4b109845d8087491ca9f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:10.908224 712 leader_election.cc:343] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:10.908636 1176 tablet_bootstrap.cc:492] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.908814 1176 ts_tablet_manager.cc:1397] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250109 20:53:10.909483 1176 raft_consensus.cc:357] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.909649 1176 raft_consensus.cc:383] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.909708 1176 raft_consensus.cc:738] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.909945 1176 consensus_queue.cc:260] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.910864 1020 tablet_service.cc:1467] Processing CreateTablet for tablet eb5d74fde6bf4e1792a18bd904d4b0b7 (DEFAULT_TABLE table=default.Table_9a175bd72fd749ac80c63aae2b1567a5 [id=2625906405cf4fdcbb83efa610789aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.913528 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5d74fde6bf4e1792a18bd904d4b0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.915179 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5056ac74d94e424aa2e7a79a8c9edd86
I20250109 20:53:10.916461 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 5056ac74d94e424aa2e7a79a8c9edd86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.917102 1176 ts_tablet_manager.cc:1428] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250109 20:53:10.917413 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 5056ac74d94e424aa2e7a79a8c9edd86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:10.917649 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 5056ac74d94e424aa2e7a79a8c9edd86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:10.920137 2802 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8328960890b142248073cec450e957d8)
I20250109 20:53:10.920261 1176 tablet_bootstrap.cc:492] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.921674 1176 tablet_bootstrap.cc:654] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.923570 2788 tablet.cc:1721] T 4f3c276fab6a4d068d64ea5d23c9a595 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:10.923940 1158 tablet_bootstrap.cc:492] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:10.924046 1158 ts_tablet_manager.cc:1397] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.310s user 0.007s sys 0.009s
I20250109 20:53:10.924198 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9d5da32f48064377ad7a7942e7122a8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.924662 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d5da32f48064377ad7a7942e7122a8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.924664 1158 raft_consensus.cc:357] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.924834 1158 raft_consensus.cc:383] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.924880 1158 raft_consensus.cc:738] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.925252 1158 consensus_queue.cc:260] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:10.931859 712 consensus_peers.cc:487] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:10.933281 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f1534c7240fc49f684af61aef783ee24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:10.934635 742 tablet_service.cc:1467] Processing CreateTablet for tablet 51f6b78fee09456e91dc2a614313d541 (DEFAULT_TABLE table=default.Table_cc91766733934db588c5f1153a4b0d0e [id=d54699a4c5b647b28f2d85c2b57b9cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.935050 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f6b78fee09456e91dc2a614313d541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.935007 886 tablet_service.cc:1467] Processing CreateTablet for tablet 77229c82aa6445f5b6bfa65d63c345ca (DEFAULT_TABLE table=default.Table_3dcc18419938438fbe7c33612df86f1b [id=f7c7437076d346e7ba0f78e7edc18adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.935551 1034 tablet_service.cc:1467] Processing CreateTablet for tablet f70a30b93ece482c98ea9cb3b85e9af8 (DEFAULT_TABLE table=default.Table_ad137ad7c05d4e4c9a3e84e8f3bb18c8 [id=c95af0fd14d84b2bbc88d14c167abdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.935587 885 tablet_service.cc:1467] Processing CreateTablet for tablet f70a30b93ece482c98ea9cb3b85e9af8 (DEFAULT_TABLE table=default.Table_ad137ad7c05d4e4c9a3e84e8f3bb18c8 [id=c95af0fd14d84b2bbc88d14c167abdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.935842 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70a30b93ece482c98ea9cb3b85e9af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.936280 884 tablet_service.cc:1467] Processing CreateTablet for tablet 39a8898aaba349f2bb111401e63da281 (DEFAULT_TABLE table=default.Table_6ee7a6b6b6c142ac8a14a4fe60c7c5b6 [id=3de085680e1846fca47b98d03384708a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.936941 1158 ts_tablet_manager.cc:1428] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250109 20:53:10.937212 1158 tablet_bootstrap.cc:492] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:10.939265 740 tablet_service.cc:1467] Processing CreateTablet for tablet 77229c82aa6445f5b6bfa65d63c345ca (DEFAULT_TABLE table=default.Table_3dcc18419938438fbe7c33612df86f1b [id=f7c7437076d346e7ba0f78e7edc18adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.939776 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77229c82aa6445f5b6bfa65d63c345ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.941108 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 77229c82aa6445f5b6bfa65d63c345ca (DEFAULT_TABLE table=default.Table_3dcc18419938438fbe7c33612df86f1b [id=f7c7437076d346e7ba0f78e7edc18adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.941460 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77229c82aa6445f5b6bfa65d63c345ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.944455 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 51f6b78fee09456e91dc2a614313d541 (DEFAULT_TABLE table=default.Table_cc91766733934db588c5f1153a4b0d0e [id=d54699a4c5b647b28f2d85c2b57b9cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.944839 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f6b78fee09456e91dc2a614313d541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.947540 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 4f3c276fab6a4d068d64ea5d23c9a595 (table default.Table_159068abc4d842c693e5ae2323d1147d [id=15c9f7d5d3ff4fa4865b55c5892421fb]). Expected version 3 got 0
I20250109 20:53:10.947816 447 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet 5ce0a941f0c14501a947a5c7e2a844e8 (table default.Table_dc8c9fd3f3844ff28cbe535e71493c94 [id=b4e108f1f95045159449a7bde6731cd6]). Expected version 1 got 0
I20250109 20:53:10.949218 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_f60b293d973d4420b9c2e4218b562a64" } new_table_name: "default.Table_444c5668ac7241f68f3553396c324b47" modify_external_catalogs: true
I20250109 20:53:10.950472 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6dde16e3825445a38efe43db92927674" } new_table_name: "default.Table_b1a1839ad80842dda6e5ed8a290f89ab" modify_external_catalogs: true
I20250109 20:53:10.951378 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b0b9646173584ad1b385b18ea22ad049"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.951916 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0b9646173584ad1b385b18ea22ad049 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.953514 752 tablet_service.cc:1467] Processing CreateTablet for tablet 39a8898aaba349f2bb111401e63da281 (DEFAULT_TABLE table=default.Table_6ee7a6b6b6c142ac8a14a4fe60c7c5b6 [id=3de085680e1846fca47b98d03384708a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.953851 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a8898aaba349f2bb111401e63da281. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.955305 887 tablet_service.cc:1467] Processing CreateTablet for tablet 61f3eb0782e04761ac39e7742431e9f3 (DEFAULT_TABLE table=default.Table_5c3902e3243943718b6fcf1b5d294445 [id=f9665c4062df47faa7d0de6fe1f2c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.957167 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61f3eb0782e04761ac39e7742431e9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.960939 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_faeda3c4ba4943eaaf3ecd5dadbcfe55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.961424 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faeda3c4ba4943eaaf3ecd5dadbcfe55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.962489 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a8898aaba349f2bb111401e63da281. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.962852 1166 ts_tablet_manager.cc:1918] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:10.962958 1166 log.cc:1198] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/d9f18f244a344272bba2bc085686b61d
I20250109 20:53:10.963209 1166 ts_tablet_manager.cc:1939] T d9f18f244a344272bba2bc085686b61d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:10.965075 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77229c82aa6445f5b6bfa65d63c345ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.965230 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70a30b93ece482c98ea9cb3b85e9af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.967841 1165 tablet_bootstrap.cc:492] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:10.967952 1165 ts_tablet_manager.cc:1397] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.000s
I20250109 20:53:10.968528 1165 raft_consensus.cc:357] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.968669 1165 raft_consensus.cc:383] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.968740 1165 raft_consensus.cc:738] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.968917 1165 consensus_queue.cc:260] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:10.969363 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0cf3e606c6e403793fe7ff8e0796efa consensus state has no leader
I20250109 20:53:10.970556 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]) successfully deleted
W20250109 20:53:10.970744 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 63a665fcf6ba43f59de8b945c5534d8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63a665fcf6ba43f59de8b945c5534d8e already in progress: creating tablet
W20250109 20:53:10.970881 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2298004317b14932bf80f39568ad73da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2298004317b14932bf80f39568ad73da already in progress: creating tablet
W20250109 20:53:10.971012 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e53966daf1ed4a67801d9ab239e17d8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e53966daf1ed4a67801d9ab239e17d8e already in progress: creating tablet
W20250109 20:53:10.971138 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5056ac74d94e424aa2e7a79a8c9edd86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5056ac74d94e424aa2e7a79a8c9edd86 already in progress: creating tablet
I20250109 20:53:10.972162 889 tablet_service.cc:1467] Processing CreateTablet for tablet 769bee026cfc4438b78f46ce91bd1b9d (DEFAULT_TABLE table=default.Table_cc5e0fd132d64de6bcd94f9f4743aa8c [id=1d95a8136cc845e991a5a95086652cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.972437 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 769bee026cfc4438b78f46ce91bd1b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.975162 882 tablet_service.cc:1467] Processing CreateTablet for tablet 917ecefedda44e02a1493d4bdc4b47e9 (DEFAULT_TABLE table=default.Table_e2c554ed27a3434b8133da5863d3ec1e [id=8918161884fc4088b8f94e748b15a49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.975507 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917ecefedda44e02a1493d4bdc4b47e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.976672 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_741be8472805423ba5df0e5dc57e2bcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.977182 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_741be8472805423ba5df0e5dc57e2bcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.978052 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bc12f5dd82b04974b89e88ba08f752b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.978415 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc12f5dd82b04974b89e88ba08f752b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.979769 881 tablet_service.cc:1467] Processing CreateTablet for tablet f0d1344bc6f14f2ca2da87aa0e6c53e6 (DEFAULT_TABLE table=default.Table_dd9b434e7fa540eca7b65552db623183 [id=9ec6db38511a41f387b4954a440128be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.980032 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d1344bc6f14f2ca2da87aa0e6c53e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.980444 883 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3a2024ed4a4ca4a2b147ad5a319505 (DEFAULT_TABLE table=default.Table_43c505c1a72e49a5b2d5fa847e9c9402 [id=947186d0c8234436b7edc96fea68ac2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.980700 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3a2024ed4a4ca4a2b147ad5a319505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.982160 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_d359b13e7edb4d83bbf14b911bc2f52d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.982517 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d359b13e7edb4d83bbf14b911bc2f52d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.984198 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7318b2f52d6846e6a5c719f130fae1c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:10.984555 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7318b2f52d6846e6a5c719f130fae1c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:10.984688 897 tablet_service.cc:1467] Processing CreateTablet for tablet 8d189db06c6441f887425237be5384af (DEFAULT_TABLE table=default.Table_7b117f0691724fb8990d5597ce687d65 [id=01ac4c6bcec8438581098f34c1bb13e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.984989 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d189db06c6441f887425237be5384af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.986455 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_34fe4c9285164667b3678c3bb7b23c03" } new_table_name: "default.Table_5944ce6d888f48ac86da4bb6c0b9a3b4" modify_external_catalogs: true
I20250109 20:53:10.987845 896 tablet_service.cc:1467] Processing CreateTablet for tablet 93466e0ad0b444ada255c80494f1967e (DEFAULT_TABLE table=default.Table_31cc0ffd3c1c44f3876b4793eae7b026 [id=1b63eaad3c4a49e590a49aa3d5bd841c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.988122 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93466e0ad0b444ada255c80494f1967e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.988444 1158 tablet_bootstrap.cc:654] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.989913 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7a758bbaa24bd089568bc1c4ad12da (DEFAULT_TABLE table=default.Table_1f395e1ff2c84808827b0a02b6575663 [id=ff5225d9736645cda75fc5b2a1bdb236]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.990274 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7a758bbaa24bd089568bc1c4ad12da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.991007 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3846effc51964fd88925fa45cbb3c22b (DEFAULT_TABLE table=default.Table_a7b76d111d3046d8890bc96cfe73deeb [id=13b674cab77940be90a984279930abb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.991267 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3846effc51964fd88925fa45cbb3c22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.991889 1176 tablet_bootstrap.cc:492] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:10.992009 1176 ts_tablet_manager.cc:1397] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.011s
I20250109 20:53:10.992962 1176 raft_consensus.cc:357] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.993218 1176 raft_consensus.cc:383] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:10.993269 1176 raft_consensus.cc:738] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:10.993466 1176 consensus_queue.cc:260] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:10.994151 892 tablet_service.cc:1467] Processing CreateTablet for tablet 34a31c174e2e406fb9336169a5752d3c (DEFAULT_TABLE table=default.Table_515514170bbe43d4ba0791fcc07c725a [id=26411a0c054f43a2ac920eacd347e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.994589 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a31c174e2e406fb9336169a5752d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.996956 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8263ebe7624c4f6da6eb5c94f4672a16 (DEFAULT_TABLE table=default.Table_f5728156039c4352b2474ae0d7148de0 [id=f013a6573a16457f9f3bca0c5b0abb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.997285 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8263ebe7624c4f6da6eb5c94f4672a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:10.998239 1176 ts_tablet_manager.cc:1428] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250109 20:53:10.998464 1176 tablet_bootstrap.cc:492] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:10.999748 888 tablet_service.cc:1467] Processing CreateTablet for tablet 19acd9c186244cc39993931d57a56066 (DEFAULT_TABLE table=default.Table_f4cefa533b6c443c8aeb122fd2787cc2 [id=a53ce58b8ccd4ad29f00632e5da1ab54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:10.999814 1176 tablet_bootstrap.cc:654] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:10.999994 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19acd9c186244cc39993931d57a56066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.000317 2754 raft_consensus.cc:491] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.000437 2754 raft_consensus.cc:513] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.000990 2754 leader_election.cc:290] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.001091 2754 raft_consensus.cc:491] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.001173 2754 raft_consensus.cc:513] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.001559 2754 leader_election.cc:290] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.002138 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcb6780422541c89343e98e02938d0e" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.002588 712 leader_election.cc:343] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.004781 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2bd365ec9d6640318cfafe056f41dbf0" } new_table_name: "default.Table_ae206aa8feab40e897d223b97a6c164b" modify_external_catalogs: true
I20250109 20:53:11.007413 1165 ts_tablet_manager.cc:1428] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20250109 20:53:11.007717 1165 tablet_bootstrap.cc:492] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.008489 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 61f3eb0782e04761ac39e7742431e9f3 (DEFAULT_TABLE table=default.Table_5c3902e3243943718b6fcf1b5d294445 [id=f9665c4062df47faa7d0de6fe1f2c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.008553 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2797e1df3d12476f89fb5155523c3935" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:11.008880 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61f3eb0782e04761ac39e7742431e9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.009018 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 769bee026cfc4438b78f46ce91bd1b9d (DEFAULT_TABLE table=default.Table_cc5e0fd132d64de6bcd94f9f4743aa8c [id=1d95a8136cc845e991a5a95086652cad]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:11.009117 712 leader_election.cc:343] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.009356 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 769bee026cfc4438b78f46ce91bd1b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.009716 1165 tablet_bootstrap.cc:654] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.010630 746 tablet_service.cc:1467] Processing CreateTablet for tablet 769bee026cfc4438b78f46ce91bd1b9d (DEFAULT_TABLE table=default.Table_cc5e0fd132d64de6bcd94f9f4743aa8c [id=1d95a8136cc845e991a5a95086652cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.010737 757 tablet_service.cc:1467] Processing CreateTablet for tablet 8263ebe7624c4f6da6eb5c94f4672a16 (DEFAULT_TABLE table=default.Table_f5728156039c4352b2474ae0d7148de0 [id=f013a6573a16457f9f3bca0c5b0abb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.010942 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 769bee026cfc4438b78f46ce91bd1b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.011148 739 tablet_service.cc:1467] Processing CreateTablet for tablet 917ecefedda44e02a1493d4bdc4b47e9 (DEFAULT_TABLE table=default.Table_e2c554ed27a3434b8133da5863d3ec1e [id=8918161884fc4088b8f94e748b15a49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.011381 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917ecefedda44e02a1493d4bdc4b47e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.012797 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8263ebe7624c4f6da6eb5c94f4672a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.012943 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 8263ebe7624c4f6da6eb5c94f4672a16 (DEFAULT_TABLE table=default.Table_f5728156039c4352b2474ae0d7148de0 [id=f013a6573a16457f9f3bca0c5b0abb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.013204 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8263ebe7624c4f6da6eb5c94f4672a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.016943 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 917ecefedda44e02a1493d4bdc4b47e9 (DEFAULT_TABLE table=default.Table_e2c554ed27a3434b8133da5863d3ec1e [id=8918161884fc4088b8f94e748b15a49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.017231 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917ecefedda44e02a1493d4bdc4b47e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.019683 1176 tablet_bootstrap.cc:492] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.019779 1176 ts_tablet_manager.cc:1397] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20250109 20:53:11.020754 743 tablet_service.cc:1467] Processing CreateTablet for tablet 34a31c174e2e406fb9336169a5752d3c (DEFAULT_TABLE table=default.Table_515514170bbe43d4ba0791fcc07c725a [id=26411a0c054f43a2ac920eacd347e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.020850 741 tablet_service.cc:1467] Processing CreateTablet for tablet 93466e0ad0b444ada255c80494f1967e (DEFAULT_TABLE table=default.Table_31cc0ffd3c1c44f3876b4793eae7b026 [id=1b63eaad3c4a49e590a49aa3d5bd841c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.020918 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet d9f18f244a344272bba2bc085686b61d (table default.Table_7ad564a8e67d49bb8a46048a1c78ddd9 [id=9ac4563802d3460f8be22b1e845b4468]) successfully deleted
I20250109 20:53:11.021023 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a31c174e2e406fb9336169a5752d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.021209 750 tablet_service.cc:1467] Processing CreateTablet for tablet 820b1da6281944648bdcdd59e532f10b (DEFAULT_TABLE table=default.Table_9541a8cce37549a2a0c33700443047f6 [id=1e4c6b825f154f4b98daebd7a45483df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.021445 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 820b1da6281944648bdcdd59e532f10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.021910 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 8d189db06c6441f887425237be5384af (DEFAULT_TABLE table=default.Table_7b117f0691724fb8990d5597ce687d65 [id=01ac4c6bcec8438581098f34c1bb13e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.022176 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d189db06c6441f887425237be5384af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.022330 745 tablet_service.cc:1467] Processing CreateTablet for tablet 3846effc51964fd88925fa45cbb3c22b (DEFAULT_TABLE table=default.Table_a7b76d111d3046d8890bc96cfe73deeb [id=13b674cab77940be90a984279930abb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.022574 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3846effc51964fd88925fa45cbb3c22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.024901 755 tablet_service.cc:1467] Processing CreateTablet for tablet f0d1344bc6f14f2ca2da87aa0e6c53e6 (DEFAULT_TABLE table=default.Table_dd9b434e7fa540eca7b65552db623183 [id=9ec6db38511a41f387b4954a440128be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.024899 1019 tablet_service.cc:1467] Processing CreateTablet for tablet f0d1344bc6f14f2ca2da87aa0e6c53e6 (DEFAULT_TABLE table=default.Table_dd9b434e7fa540eca7b65552db623183 [id=9ec6db38511a41f387b4954a440128be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.025043 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 34a31c174e2e406fb9336169a5752d3c (DEFAULT_TABLE table=default.Table_515514170bbe43d4ba0791fcc07c725a [id=26411a0c054f43a2ac920eacd347e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.025163 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d1344bc6f14f2ca2da87aa0e6c53e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.025260 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93466e0ad0b444ada255c80494f1967e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.025260 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3a2024ed4a4ca4a2b147ad5a319505 (DEFAULT_TABLE table=default.Table_43c505c1a72e49a5b2d5fa847e9c9402 [id=947186d0c8234436b7edc96fea68ac2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.024899 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 93466e0ad0b444ada255c80494f1967e (DEFAULT_TABLE table=default.Table_31cc0ffd3c1c44f3876b4793eae7b026 [id=1b63eaad3c4a49e590a49aa3d5bd841c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.025468 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7a758bbaa24bd089568bc1c4ad12da (DEFAULT_TABLE table=default.Table_1f395e1ff2c84808827b0a02b6575663 [id=ff5225d9736645cda75fc5b2a1bdb236]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.024924 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 820b1da6281944648bdcdd59e532f10b (DEFAULT_TABLE table=default.Table_9541a8cce37549a2a0c33700443047f6 [id=1e4c6b825f154f4b98daebd7a45483df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.025643 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 3846effc51964fd88925fa45cbb3c22b (DEFAULT_TABLE table=default.Table_a7b76d111d3046d8890bc96cfe73deeb [id=13b674cab77940be90a984279930abb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.026083 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7a758bbaa24bd089568bc1c4ad12da (DEFAULT_TABLE table=default.Table_1f395e1ff2c84808827b0a02b6575663 [id=ff5225d9736645cda75fc5b2a1bdb236]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.026468 747 tablet_service.cc:1467] Processing CreateTablet for tablet 19acd9c186244cc39993931d57a56066 (DEFAULT_TABLE table=default.Table_f4cefa533b6c443c8aeb122fd2787cc2 [id=a53ce58b8ccd4ad29f00632e5da1ab54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.025270 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a31c174e2e406fb9336169a5752d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.026960 746 tablet_service.cc:1467] Processing CreateTablet for tablet 8d189db06c6441f887425237be5384af (DEFAULT_TABLE table=default.Table_7b117f0691724fb8990d5597ce687d65 [id=01ac4c6bcec8438581098f34c1bb13e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.027223 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93466e0ad0b444ada255c80494f1967e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.027402 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7a758bbaa24bd089568bc1c4ad12da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.028838 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 820b1da6281944648bdcdd59e532f10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.032454 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3846effc51964fd88925fa45cbb3c22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.033774 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d1344bc6f14f2ca2da87aa0e6c53e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.034139 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7a758bbaa24bd089568bc1c4ad12da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.035005 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d189db06c6441f887425237be5384af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.036939 2749 raft_consensus.cc:491] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.037043 2749 raft_consensus.cc:513] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.037586 2749 leader_election.cc:290] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:11.038398 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049321d28ab241138247dcefa0411f54" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:11.038904 990 leader_election.cc:343] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.039124 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19acd9c186244cc39993931d57a56066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.039608 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a82e1240e6aa4513b7008ddeb3dff288" } modify_external_catalogs: true
I20250109 20:53:11.039757 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_a82e1240e6aa4513b7008ddeb3dff288" } modify_external_catalogs: true
I20250109 20:53:11.040447 742 tablet_service.cc:1467] Processing CreateTablet for tablet 61f3eb0782e04761ac39e7742431e9f3 (DEFAULT_TABLE table=default.Table_5c3902e3243943718b6fcf1b5d294445 [id=f9665c4062df47faa7d0de6fe1f2c382]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.040733 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61f3eb0782e04761ac39e7742431e9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.041101 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049321d28ab241138247dcefa0411f54" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.041954 989 leader_election.cc:343] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.042042 989 leader_election.cc:304] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.043354 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3f23e65ad9c144d29bdb880a12dbfc73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.043763 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f23e65ad9c144d29bdb880a12dbfc73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.045774 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3a2024ed4a4ca4a2b147ad5a319505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.049230 1166 tablet_replica.cc:331] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.049401 1166 raft_consensus.cc:2238] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.049544 1166 raft_consensus.cc:2267] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.050015 1166 ts_tablet_manager.cc:1905] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.052788 2749 raft_consensus.cc:2743] T 049321d28ab241138247dcefa0411f54 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.020448 1176 raft_consensus.cc:357] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.058264 1158 tablet_bootstrap.cc:492] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.058373 1158 ts_tablet_manager.cc:1397] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.000s
I20250109 20:53:11.058468 1176 raft_consensus.cc:383] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.058566 1176 raft_consensus.cc:738] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.058796 1176 consensus_queue.cc:260] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.059015 1158 raft_consensus.cc:357] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.059182 1158 raft_consensus.cc:383] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.059238 1158 raft_consensus.cc:738] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.059427 1158 consensus_queue.cc:260] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.059540 1176 ts_tablet_manager.cc:1428] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20250109 20:53:11.059841 1176 tablet_bootstrap.cc:492] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.061260 1176 tablet_bootstrap.cc:654] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.061677 891 tablet_service.cc:1467] Processing CreateTablet for tablet 046ba213bde34e35b144d3177c62d79f (DEFAULT_TABLE table=default.Table_34fe4c9285164667b3678c3bb7b23c03 [id=c327ada8de674629b7f5eda3306422fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.062005 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 046ba213bde34e35b144d3177c62d79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.064828 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_19aea38f8d4547399b10df16cdb1e54e" } new_table_name: "default.Table_bc4981434b6743fc93063e84ec2ed645" modify_external_catalogs: true
I20250109 20:53:11.061677 894 tablet_service.cc:1467] Processing CreateTablet for tablet bd7f542fe5c94de99c369e291d39304d (DEFAULT_TABLE table=default.Table_1e5d4f2960e444ecbc9a20662e2bf4b4 [id=a03025f9e05f42c9b1dd218ff1a8876b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.065176 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd7f542fe5c94de99c369e291d39304d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.073772 743 tablet_service.cc:1467] Processing CreateTablet for tablet fec31c6ea3b340a4a89be5069ee5ec35 (DEFAULT_TABLE table=default.Table_f3b977ba7d594fb7aeb874d8b5e7e423 [id=76a8ccb56655408d920d86ca75c019ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.073772 1029 tablet_service.cc:1467] Processing CreateTablet for tablet fec31c6ea3b340a4a89be5069ee5ec35 (DEFAULT_TABLE table=default.Table_f3b977ba7d594fb7aeb874d8b5e7e423 [id=76a8ccb56655408d920d86ca75c019ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.074100 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec31c6ea3b340a4a89be5069ee5ec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.074120 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec31c6ea3b340a4a89be5069ee5ec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.074848 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet c83b61e879fc4979a85c872f2e6ab88c with delete_type TABLET_DATA_DELETED (replaced by tablet c3d9f992800e4a41a7af7e2751214eaa) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.076701 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 046ba213bde34e35b144d3177c62d79f (DEFAULT_TABLE table=default.Table_34fe4c9285164667b3678c3bb7b23c03 [id=c327ada8de674629b7f5eda3306422fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.077039 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 046ba213bde34e35b144d3177c62d79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.077634 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 0721e92141fe4be482391088bb55c592 (DEFAULT_TABLE table=default.Table_e01393f526ac40c79f01d50fc724ef49 [id=2db0a92c98b845b0afbea35b957d2cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.077891 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0721e92141fe4be482391088bb55c592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.078506 750 tablet_service.cc:1467] Processing CreateTablet for tablet bd7f542fe5c94de99c369e291d39304d (DEFAULT_TABLE table=default.Table_1e5d4f2960e444ecbc9a20662e2bf4b4 [id=a03025f9e05f42c9b1dd218ff1a8876b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.078650 741 tablet_service.cc:1467] Processing CreateTablet for tablet 97e0335cef0c432fbe2a4002e9876c52 (DEFAULT_TABLE table=default.Table_0f0e855cab8543789fddd27edca161f6 [id=50ec672430f44c4fad14259b5daa27a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.078780 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd7f542fe5c94de99c369e291d39304d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.079007 747 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9be110cfdb4126864752834bd834f5 (DEFAULT_TABLE table=default.Table_2d0128d7e559487f8176051023f82704 [id=8a7535efb30a4bdf9097ea74d9340dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.079254 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e9be110cfdb4126864752834bd834f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.078506 754 tablet_service.cc:1467] Processing CreateTablet for tablet 046ba213bde34e35b144d3177c62d79f (DEFAULT_TABLE table=default.Table_34fe4c9285164667b3678c3bb7b23c03 [id=c327ada8de674629b7f5eda3306422fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.080552 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 046ba213bde34e35b144d3177c62d79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.081037 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9be110cfdb4126864752834bd834f5 (DEFAULT_TABLE table=default.Table_2d0128d7e559487f8176051023f82704 [id=8a7535efb30a4bdf9097ea74d9340dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.081306 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e9be110cfdb4126864752834bd834f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.081463 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 19acd9c186244cc39993931d57a56066 (DEFAULT_TABLE table=default.Table_f4cefa533b6c443c8aeb122fd2787cc2 [id=a53ce58b8ccd4ad29f00632e5da1ab54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.081725 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19acd9c186244cc39993931d57a56066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.084386 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e0335cef0c432fbe2a4002e9876c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.084697 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 97e0335cef0c432fbe2a4002e9876c52 (DEFAULT_TABLE table=default.Table_0f0e855cab8543789fddd27edca161f6 [id=50ec672430f44c4fad14259b5daa27a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.084981 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e0335cef0c432fbe2a4002e9876c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.089072 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba1531d1d3a4253bb8ed1bd4141605a (DEFAULT_TABLE table=default.Table_d6579e9bf91a4d5a93b219c3e20784ea [id=806874da09be41c0a838ad3c3d22db68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.089355 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba1531d1d3a4253bb8ed1bd4141605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.090339 1158 ts_tablet_manager.cc:1428] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250109 20:53:11.093133 1023 tablet_service.cc:1467] Processing CreateTablet for tablet bd7f542fe5c94de99c369e291d39304d (DEFAULT_TABLE table=default.Table_1e5d4f2960e444ecbc9a20662e2bf4b4 [id=a03025f9e05f42c9b1dd218ff1a8876b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.093384 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd7f542fe5c94de99c369e291d39304d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.094156 1176 tablet_bootstrap.cc:492] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.094269 1176 ts_tablet_manager.cc:1397] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.005s
I20250109 20:53:11.095082 1158 tablet_bootstrap.cc:492] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.096310 1158 tablet_bootstrap.cc:654] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.098080 1176 raft_consensus.cc:357] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.098245 1176 raft_consensus.cc:383] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.098290 1176 raft_consensus.cc:738] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.098452 1176 consensus_queue.cc:260] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.098978 1176 ts_tablet_manager.cc:1428] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250109 20:53:11.099196 1176 tablet_bootstrap.cc:492] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.099713 739 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3a2024ed4a4ca4a2b147ad5a319505 (DEFAULT_TABLE table=default.Table_43c505c1a72e49a5b2d5fa847e9c9402 [id=947186d0c8234436b7edc96fea68ac2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.100023 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3a2024ed4a4ca4a2b147ad5a319505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.100589 1176 tablet_bootstrap.cc:654] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.101263 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba1531d1d3a4253bb8ed1bd4141605a (DEFAULT_TABLE table=default.Table_d6579e9bf91a4d5a93b219c3e20784ea [id=806874da09be41c0a838ad3c3d22db68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.101552 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba1531d1d3a4253bb8ed1bd4141605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.104959 745 tablet_service.cc:1467] Processing CreateTablet for tablet 0721e92141fe4be482391088bb55c592 (DEFAULT_TABLE table=default.Table_e01393f526ac40c79f01d50fc724ef49 [id=2db0a92c98b845b0afbea35b957d2cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.105283 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0721e92141fe4be482391088bb55c592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.108186 2805 raft_consensus.cc:491] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.108300 2805 raft_consensus.cc:513] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.109213 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3293f49190684b72a5c41031e621bd36" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.109680 712 leader_election.cc:343] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.110038 2805 leader_election.cc:290] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.113865 888 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba1531d1d3a4253bb8ed1bd4141605a (DEFAULT_TABLE table=default.Table_d6579e9bf91a4d5a93b219c3e20784ea [id=806874da09be41c0a838ad3c3d22db68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.114264 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba1531d1d3a4253bb8ed1bd4141605a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.120150 880 tablet_service.cc:1467] Processing CreateTablet for tablet 97e0335cef0c432fbe2a4002e9876c52 (DEFAULT_TABLE table=default.Table_0f0e855cab8543789fddd27edca161f6 [id=50ec672430f44c4fad14259b5daa27a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.120481 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e0335cef0c432fbe2a4002e9876c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.124514 1165 tablet_bootstrap.cc:492] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.124648 1165 ts_tablet_manager.cc:1397] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.117s user 0.003s sys 0.000s
I20250109 20:53:11.124961 892 tablet_service.cc:1467] Processing CreateTablet for tablet fec31c6ea3b340a4a89be5069ee5ec35 (DEFAULT_TABLE table=default.Table_f3b977ba7d594fb7aeb874d8b5e7e423 [id=76a8ccb56655408d920d86ca75c019ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.125294 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec31c6ea3b340a4a89be5069ee5ec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.125600 1165 raft_consensus.cc:357] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.125771 1165 raft_consensus.cc:383] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.125823 1165 raft_consensus.cc:738] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.126106 1165 consensus_queue.cc:260] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.126803 1165 ts_tablet_manager.cc:1428] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:11.127283 1157 log.cc:1198] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2769e5906c604bd5b14df79db765ec61
I20250109 20:53:11.127759 1157 ts_tablet_manager.cc:1939] T 2769e5906c604bd5b14df79db765ec61 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:11.129089 1157 tablet_replica.cc:331] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:11.129145 882 tablet_service.cc:1467] Processing CreateTablet for tablet 820b1da6281944648bdcdd59e532f10b (DEFAULT_TABLE table=default.Table_9541a8cce37549a2a0c33700443047f6 [id=1e4c6b825f154f4b98daebd7a45483df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.129334 1157 raft_consensus.cc:2238] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:11.129750 1165 tablet_bootstrap.cc:492] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.130240 884 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9be110cfdb4126864752834bd834f5 (DEFAULT_TABLE table=default.Table_2d0128d7e559487f8176051023f82704 [id=8a7535efb30a4bdf9097ea74d9340dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.131314 1165 tablet_bootstrap.cc:654] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.136981 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e9be110cfdb4126864752834bd834f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.137425 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2769e5906c604bd5b14df79db765ec61 (table default.Table_f59d59762b8b4f55b3123c4df5bfd12e [id=3c9451140e48432f8591841d4dfd8751]) successfully deleted
I20250109 20:53:11.138051 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 820b1da6281944648bdcdd59e532f10b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:11.138557 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet d0a5a92a65dc4547881bf914b88c9b84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0a5a92a65dc4547881bf914b88c9b84 already in progress: creating tablet
I20250109 20:53:11.142778 885 tablet_service.cc:1467] Processing CreateTablet for tablet 0721e92141fe4be482391088bb55c592 (DEFAULT_TABLE table=default.Table_e01393f526ac40c79f01d50fc724ef49 [id=2db0a92c98b845b0afbea35b957d2cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.143530 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0721e92141fe4be482391088bb55c592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.148111 895 tablet_service.cc:1467] Processing CreateTablet for tablet 51f6b78fee09456e91dc2a614313d541 (DEFAULT_TABLE table=default.Table_cc91766733934db588c5f1153a4b0d0e [id=d54699a4c5b647b28f2d85c2b57b9cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.148654 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51f6b78fee09456e91dc2a614313d541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.154649 749 tablet_service.cc:1467] Processing CreateTablet for tablet f70a30b93ece482c98ea9cb3b85e9af8 (DEFAULT_TABLE table=default.Table_ad137ad7c05d4e4c9a3e84e8f3bb18c8 [id=c95af0fd14d84b2bbc88d14c167abdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.155045 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f70a30b93ece482c98ea9cb3b85e9af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.161469 1166 ts_tablet_manager.cc:1918] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:11.161589 1166 log.cc:1198] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/70b48900f57f47128e31b27275b1489c
I20250109 20:53:11.161945 1166 ts_tablet_manager.cc:1939] T 70b48900f57f47128e31b27275b1489c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.162328 1176 tablet_bootstrap.cc:492] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.162464 1176 ts_tablet_manager.cc:1397] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.011s
I20250109 20:53:11.163707 1176 raft_consensus.cc:357] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.163908 1176 raft_consensus.cc:383] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.163970 1176 raft_consensus.cc:738] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.164189 1176 consensus_queue.cc:260] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.164894 1176 ts_tablet_manager.cc:1428] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:11.165203 1176 tablet_bootstrap.cc:492] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.165196 1158 tablet_bootstrap.cc:492] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.165320 1158 ts_tablet_manager.cc:1397] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20250109 20:53:11.166867 1176 tablet_bootstrap.cc:654] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.165987 1158 raft_consensus.cc:357] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.168154 1158 raft_consensus.cc:383] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.168213 1158 raft_consensus.cc:738] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.169994 1158 consensus_queue.cc:260] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.173488 1158 ts_tablet_manager.cc:1428] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250109 20:53:11.174365 1158 tablet_bootstrap.cc:492] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.176110 1158 tablet_bootstrap.cc:654] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.177295 2633 raft_consensus.cc:491] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.177436 2633 raft_consensus.cc:513] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.178085 2633 leader_election.cc:290] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.179085 1165 tablet_bootstrap.cc:492] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.179224 1165 ts_tablet_manager.cc:1397] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
W20250109 20:53:11.179602 854 leader_election.cc:343] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 006475691733470f8a0468365972dfd2
I20250109 20:53:11.179937 1165 raft_consensus.cc:357] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.180102 1165 raft_consensus.cc:383] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.180166 1165 raft_consensus.cc:738] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
W20250109 20:53:11.180440 854 leader_election.cc:343] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 006475691733470f8a0468365972dfd2
I20250109 20:53:11.180377 1165 consensus_queue.cc:260] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.180531 854 leader_election.cc:304] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.180868 1165 ts_tablet_manager.cc:1428] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:11.180994 2633 raft_consensus.cc:2743] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.181092 1165 tablet_bootstrap.cc:492] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.182492 1165 tablet_bootstrap.cc:654] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.183827 2754 raft_consensus.cc:491] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.183940 2754 raft_consensus.cc:513] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.184459 2754 leader_election.cc:290] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.184960 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af9563f6efd47059caac7c86c41d43f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.185379 712 leader_election.cc:343] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:11.196344 2788 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.272s user 0.000s sys 0.000s
I20250109 20:53:07.083711 2167 rpcz_store.cc:276] 0109 20:52:59.954586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:52:59.954656 (+ 70us) service_pool.cc:225] Handling call
0109 20:53:06.351427 (+6396771us) raft_consensus.cc:1466] Updating replica for 425 ops
0109 20:53:06.480464 (+129037us) raft_consensus.cc:1529] Early marking committed up to index 0
0109 20:53:06.480475 (+ 11us) raft_consensus.cc:1534] Triggering prepare for 425 ops
0109 20:53:06.481800 (+ 1325us) write_op.cc:270] Start()
0109 20:53:06.481853 (+ 53us) write_op.cc:276] Timestamp: P: 1736455972849819 usec, L: 0
0109 20:53:06.514530 (+ 32677us) write_op.cc:270] Start()
0109 20:53:06.514561 (+ 31us) write_op.cc:276] Timestamp: P: 1736455972850769 usec, L: 0
0109 20:53:06.514784 (+ 223us) write_op.cc:270] Start()
0109 20:53:06.514823 (+ 39us) write_op.cc:276] Timestamp: P: 1736455972857068 usec, L: 0
0109 20:53:06.515139 (+ 316us) write_op.cc:270] Start()
0109 20:53:06.515162 (+ 23us) write_op.cc:276] Timestamp: P: 1736455972923236 usec, L: 0
0109 20:53:06.515313 (+ 151us) write_op.cc:270] Start()
0109 20:53:06.515345 (+ 32us) write_op.cc:276] Timestamp: P: 1736455972924370 usec, L: 0
0109 20:53:06.515486 (+ 141us) write_op.cc:270] Start()
0109 20:53:06.515505 (+ 19us) write_op.cc:276] Timestamp: P: 1736455972925355 usec, L: 0
0109 20:53:06.515631 (+ 126us) write_op.cc:270] Start()
0109 20:53:06.515674 (+ 43us) write_op.cc:276] Timestamp: P: 1736455972926154 usec, L: 0
0109 20:53:06.515832 (+ 158us) write_op.cc:270] Start()
0109 20:53:06.515849 (+ 17us) write_op.cc:276] Timestamp: P: 1736455972926893 usec, L: 0
0109 20:53:06.515985 (+ 136us) write_op.cc:270] Start()
0109 20:53:06.516001 (+ 16us) write_op.cc:276] Timestamp: P: 1736455972929849 usec, L: 0
0109 20:53:06.516106 (+ 105us) write_op.cc:270] Start()
0109 20:53:06.516122 (+ 16us) write_op.cc:276] Timestamp: P: 1736455972951754 usec, L: 0
0109 20:53:06.516256 (+ 134us) write_op.cc:270] Start()
0109 20:53:06.516273 (+ 17us) write_op.cc:276] Timestamp: P: 1736455972958251 usec, L: 0
0109 20:53:06.516399 (+ 126us) write_op.cc:270] Start()
0109 20:53:06.516416 (+ 17us) write_op.cc:276] Timestamp: P: 1736455972989395 usec, L: 0
0109 20:53:06.516520 (+ 104us) write_op.cc:270] Start()
0109 20:53:06.516535 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973022473 usec, L: 0
0109 20:53:06.516654 (+ 119us) write_op.cc:270] Start()
0109 20:53:06.516670 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973023040 usec, L: 0
0109 20:53:06.516844 (+ 174us) write_op.cc:270] Start()
0109 20:53:06.516862 (+ 18us) write_op.cc:276] Timestamp: P: 1736455973023475 usec, L: 0
0109 20:53:06.517009 (+ 147us) write_op.cc:270] Start()
0109 20:53:06.517025 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973024010 usec, L: 0
0109 20:53:06.517133 (+ 108us) write_op.cc:270] Start()
0109 20:53:06.517157 (+ 24us) write_op.cc:276] Timestamp: P: 1736455973024543 usec, L: 0
0109 20:53:06.517265 (+ 108us) write_op.cc:270] Start()
0109 20:53:06.517278 (+ 13us) write_op.cc:276] Timestamp: P: 1736455973025121 usec, L: 0
0109 20:53:06.517367 (+ 89us) write_op.cc:270] Start()
0109 20:53:06.517381 (+ 14us) write_op.cc:276] Timestamp: P: 1736455973053446 usec, L: 0
0109 20:53:06.517494 (+ 113us) write_op.cc:270] Start()
0109 20:53:06.517508 (+ 14us) write_op.cc:276] Timestamp: P: 1736455973202634 usec, L: 0
0109 20:53:06.517598 (+ 90us) write_op.cc:270] Start()
0109 20:53:06.517611 (+ 13us) write_op.cc:276] Timestamp: P: 1736455973209669 usec, L: 0
0109 20:53:06.517693 (+ 82us) write_op.cc:270] Start()
0109 20:53:06.517706 (+ 13us) write_op.cc:276] Timestamp: P: 1736455973210258 usec, L: 0
0109 20:53:06.517818 (+ 112us) write_op.cc:270] Start()
0109 20:53:06.517833 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973210809 usec, L: 0
0109 20:53:06.517934 (+ 101us) write_op.cc:270] Start()
0109 20:53:06.517947 (+ 13us) write_op.cc:276] Timestamp: P: 1736455973211254 usec, L: 0
0109 20:53:06.518060 (+ 113us) write_op.cc:270] Start()
0109 20:53:06.518076 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973211741 usec, L: 0
0109 20:53:06.518184 (+ 108us) write_op.cc:270] Start()
0109 20:53:06.518198 (+ 14us) write_op.cc:276] Timestamp: P: 1736455973212388 usec, L: 0
0109 20:53:06.518287 (+ 89us) write_op.cc:270] Start()
0109 20:53:06.518302 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973217079 usec, L: 0
0109 20:53:06.518407 (+ 105us) write_op.cc:270] Start()
0109 20:53:06.518422 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973224547 usec, L: 0
0109 20:53:06.518525 (+ 103us) write_op.cc:270] Start()
0109 20:53:06.518547 (+ 22us) write_op.cc:276] Timestamp: P: 1736455973260443 usec, L: 0
0109 20:53:06.518639 (+ 92us) write_op.cc:270] Start()
0109 20:53:06.518653 (+ 14us) write_op.cc:276] Timestamp: P: 1736455973261374 usec, L: 0
0109 20:53:06.518769 (+ 116us) write_op.cc:270] Start()
0109 20:53:06.518782 (+ 13us) write_op.cc:276] Timestamp: P: 1736455973262297 usec, L: 0
0109 20:53:06.518877 (+ 95us) write_op.cc:270] Start()
0109 20:53:06.518894 (+ 17us) write_op.cc:276] Timestamp: P: 1736455973263967 usec, L: 0
0109 20:53:06.519005 (+ 111us) write_op.cc:270] Start()
0109 20:53:06.519018 (+ 13us) write_op.cc:276] Timestamp: P: 1736455973273396 usec, L: 0
0109 20:53:06.522885 (+ 3867us) write_op.cc:270] Start()
0109 20:53:06.522907 (+ 22us) write_op.cc:276] Timestamp: P: 1736455973275862 usec, L: 0
0109 20:53:06.523162 (+ 255us) write_op.cc:270] Start()
0109 20:53:06.523186 (+ 24us) write_op.cc:276] Timestamp: P: 1736455973298130 usec, L: 0
0109 20:53:06.523324 (+ 138us) write_op.cc:270] Start()
0109 20:53:06.523341 (+ 17us) write_op.cc:276] Timestamp: P: 1736455973298848 usec, L: 0
0109 20:53:06.523449 (+ 108us) write_op.cc:270] Start()
0109 20:53:06.525820 (+ 2371us) write_op.cc:276] Timestamp: P: 1736455973537466 usec, L: 0
0109 20:53:06.526026 (+ 206us) write_op.cc:270] Start()
0109 20:53:06.526048 (+ 22us) write_op.cc:276] Timestamp: P: 1736455973538092 usec, L: 0
0109 20:53:06.526182 (+ 134us) write_op.cc:270] Start()
0109 20:53:06.526201 (+ 19us) write_op.cc:276] Timestamp: P: 1736455973546338 usec, L: 0
0109 20:53:06.526311 (+ 110us) write_op.cc:270] Start()
0109 20:53:06.526327 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973546957 usec, L: 0
0109 20:53:06.526435 (+ 108us) write_op.cc:270] Start()
0109 20:53:06.526486 (+ 51us) write_op.cc:276] Timestamp: P: 1736455973547500 usec, L: 0
0109 20:53:06.526604 (+ 118us) write_op.cc:270] Start()
0109 20:53:06.526621 (+ 17us) write_op.cc:276] Timestamp: P: 1736455973548098 usec, L: 0
0109 20:53:06.526748 (+ 127us) write_op.cc:270] Start()
0109 20:53:06.526763 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973556483 usec, L: 0
0109 20:53:06.526875 (+ 112us) write_op.cc:270] Start()
0109 20:53:06.526890 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973557000 usec, L: 0
0109 20:53:06.526994 (+ 104us) write_op.cc:270] Start()
0109 20:53:06.527010 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973588768 usec, L: 0
0109 20:53:06.527111 (+ 101us) write_op.cc:270] Start()
0109 20:53:06.527126 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973589264 usec, L: 0
0109 20:53:06.527243 (+ 117us) write_op.cc:270] Start()
0109 20:53:06.527258 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973589728 usec, L: 0
0109 20:53:06.527354 (+ 96us) write_op.cc:270] Start()
0109 20:53:06.527369 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973590103 usec, L: 0
0109 20:53:06.527496 (+ 127us) write_op.cc:270] Start()
0109 20:53:06.527512 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973590460 usec, L: 0
0109 20:53:06.527613 (+ 101us) write_op.cc:270] Start()
0109 20:53:06.527628 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973593222 usec, L: 0
0109 20:53:06.527794 (+ 166us) write_op.cc:270] Start()
0109 20:53:06.527810 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973614925 usec, L: 0
0109 20:53:06.527949 (+ 139us) write_op.cc:270] Start()
0109 20:53:06.527968 (+ 19us) write_op.cc:276] Timestamp: P: 1736455973616458 usec, L: 0
0109 20:53:06.528070 (+ 102us) write_op.cc:270] Start()
0109 20:53:06.528085 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973616946 usec, L: 0
0109 20:53:06.528204 (+ 119us) write_op.cc:270] Start()
0109 20:53:06.528220 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973617317 usec, L: 0
0109 20:53:06.528328 (+ 108us) write_op.cc:270] Start()
0109 20:53:06.528343 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973617692 usec, L: 0
0109 20:53:06.528450 (+ 107us) write_op.cc:270] Start()
0109 20:53:06.528465 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973773713 usec, L: 0
0109 20:53:06.528571 (+ 106us) write_op.cc:270] Start()
0109 20:53:06.528586 (+ 15us) write_op.cc:276] Timestamp: P: 1736455973783952 usec, L: 0
0109 20:53:06.536750 (+ 8164us) write_op.cc:270] Start()
0109 20:53:06.536786 (+ 36us) write_op.cc:276] Timestamp: P: 1736455973832051 usec, L: 0
0109 20:53:06.536972 (+ 186us) write_op.cc:270] Start()
0109 20:53:06.536993 (+ 21us) write_op.cc:276] Timestamp: P: 1736455973845374 usec, L: 0
0109 20:53:06.537159 (+ 166us) write_op.cc:270] Start()
0109 20:53:06.537201 (+ 42us) write_op.cc:276] Timestamp: P: 1736455973858028 usec, L: 0
0109 20:53:06.537336 (+ 135us) write_op.cc:270] Start()
0109 20:53:06.537356 (+ 20us) write_op.cc:276] Timestamp: P: 1736455973860299 usec, L: 0
0109 20:53:06.537473 (+ 117us) write_op.cc:270] Start()
0109 20:53:06.537490 (+ 17us) write_op.cc:276] Timestamp: P: 1736455973869188 usec, L: 0
0109 20:53:06.537639 (+ 149us) write_op.cc:270] Start()
0109 20:53:06.537657 (+ 18us) write_op.cc:276] Timestamp: P: 1736455973940299 usec, L: 0
0109 20:53:06.537776 (+ 119us) write_op.cc:270] Start()
0109 20:53:06.537792 (+ 16us) write_op.cc:276] Timestamp: P: 1736455973991200 usec, L: 0
0109 20:53:06.537998 (+ 206us) write_op.cc:270] Start()
0109 20:53:06.538018 (+ 20us) write_op.cc:276] Timestamp: P: 1736455973992075 usec, L: 0
0109 20:53:06.538136 (+ 118us) write_op.cc:270] Start()
0109 20:53:06.538156 (+ 20us) write_op.cc:276] Timestamp: P: 1736455973997882 usec, L: 0
0109 20:53:06.538401 (+ 245us) write_op.cc:270] Start()
0109 20:53:06.538422 (+ 21us) write_op.cc:276] Timestamp: P: 1736455973998453 usec, L: 0
0109 20:53:06.538549 (+ 127us) write_op.cc:270] Start()
0109 20:53:06.538567 (+ 18us) write_op.cc:276] Timestamp: P: 1736455973999035 usec, L: 0
0109 20:53:06.538676 (+ 109us) write_op.cc:270] Start()
0109 20:53:06.538694 (+ 18us) write_op.cc:276] Timestamp: P: 1736455974010037 usec, L: 0
0109 20:53:06.538838 (+ 144us) write_op.cc:270] Start()
0109 20:53:06.538857 (+ 19us) write_op.cc:276] Timestamp: P: 1736455974011688 usec, L: 0
0109 20:53:06.538988 (+ 131us) write_op.cc:270] Start()
0109 20:53:06.539005 (+ 17us) write_op.cc:276] Timestamp: P: 1736455974024829 usec, L: 0
0109 20:53:06.539122 (+ 117us) write_op.cc:270] Start()
0109 20:53:06.539140 (+ 18us) write_op.cc:276] Timestamp: P: 1736455974025651 usec, L: 0
0109 20:53:06.539277 (+ 137us) write_op.cc:270] Start()
0109 20:53:06.539294 (+ 17us) write_op.cc:276] Timestamp: P: 1736455974026385 usec, L: 0
0109 20:53:06.539417 (+ 123us) write_op.cc:270] Start()
0109 20:53:06.539435 (+ 18us) write_op.cc:276] Timestamp: P: 1736455974027186 usec, L: 0
0109 20:53:06.539556 (+ 121us) write_op.cc:270] Start()
0109 20:53:06.539578 (+ 22us) write_op.cc:276] Timestamp: P: 1736455974027808 usec, L: 0
0109 20:53:06.539731 (+ 153us) write_op.cc:270] Start()
0109 20:53:06.539750 (+ 19us) write_op.cc:276] Timestamp: P: 1736455974046803 usec, L: 0
0109 20:53:06.539901 (+ 151us) write_op.cc:270] Start()
0109 20:53:06.539922 (+ 21us) write_op.cc:276] Timestamp: P: 1736455974055532 usec, L: 0
0109 20:53:06.540049 (+ 127us) write_op.cc:270] Start()
0109 20:53:06.540068 (+ 19us) write_op.cc:276] Timestamp: P: 1736455974056360 usec, L: 0
0109 20:53:06.540204 (+ 136us) write_op.cc:270] Start()
0109 20:53:06.540238 (+ 34us) write_op.cc:276] Timestamp: P: 1736455974056994 usec, L: 0
0109 20:53:06.540370 (+ 132us) write_op.cc:270] Start()
0109 20:53:06.540386 (+ 16us) write_op.cc:276] Timestamp: P: 1736455974097947 usec, L: 0
0109 20:53:06.540519 (+ 133us) write_op.cc:270] Start()
0109 20:53:06.540536 (+ 17us) write_op.cc:276] Timestamp: P: 1736455974143222 usec, L: 0
0109 20:53:06.540649 (+ 113us) write_op.cc:270] Start()
0109 20:53:06.540665 (+ 16us) write_op.cc:276] Timestamp: P: 1736455974215203 usec, L: 0
0109 20:53:06.540794 (+ 129us) write_op.cc:270] Start()
0109 20:53:06.540810 (+ 16us) write_op.cc:276] Timestamp: P: 1736455974216620 usec, L: 0
0109 20:53:06.540948 (+ 138us) write_op.cc:270] Start()
0109 20:53:06.540966 (+ 18us) write_op.cc:276] Timestamp: P: 1736455974242845 usec, L: 0
0109 20:53:06.541077 (+ 111us) write_op.cc:270] Start()
0109 20:53:06.541092 (+ 15us) write_op.cc:276] Timestamp: P: 1736455974243461 usec, L: 0
0109 20:53:06.541200 (+ 108us) write_op.cc:270] Start()
0109 20:53:06.541216 (+ 16us) write_op.cc:276] Timestamp: P: 1736455974433635 usec, L: 0
0109 20:53:06.541330 (+ 114us) write_op.cc:270] Start()
0109 20:53:06.541490 (+ 160us) write_op.cc:276] Timestamp: P: 1736455974444083 usec, L: 0
0109 20:53:06.541642 (+ 152us) write_op.cc:270] Start()
0109 20:53:06.541659 (+ 17us) write_op.cc:276] Timestamp: P: 1736455974445563 usec, L: 0
0109 20:53:06.541784 (+ 125us) write_op.cc:270] Start()
0109 20:53:06.541800 (+ 16us) write_op.cc:276] Timestamp: P: 1736455974446601 usec, L: 0
0109 20:53:06.541927 (+ 127us) write_op.cc:270] Start()
0109 20:53:06.541943 (+ 16us) write_op.cc:276] Timestamp: P: 1736455974448421 usec, L: 0
0109 20:53:06.542075 (+ 132us) write_op.cc:270] Start()
0109 20:53:06.542092 (+ 17us) write_op.cc:276] Timestamp: P: 1736455974453523 usec, L: 0
0109 20:53:06.542207 (+ 115us) write_op.cc:270] Start()
0109 20:53:06.542227 (+ 20us) write_op.cc:276] Timestamp: P: 1736455974509545 usec, L: 0
0109 20:53:06.542355 (+ 128us) write_op.cc:270] Start()
0109 20:53:06.542375 (+ 20us) write_op.cc:276] Timestamp: P: 1736455974510959 usec, L: 0
0109 20:53:06.544335 (+ 1960us) write_op.cc:270] Start()
0109 20:53:06.544380 (+ 45us) write_op.cc:276] Timestamp: P: 1736455974512004 usec, L: 0
0109 20:53:06.544709 (+ 329us) write_op.cc:270] Start()
0109 20:53:06.544764 (+ 55us) write_op.cc:276] Timestamp: P: 1736455974526221 usec, L: 0
0109 20:53:06.544957 (+ 193us) write_op.cc:270] Start()
0109 20:53:06.544988 (+ 31us) write_op.cc:276] Timestamp: P: 1736455974533666 usec, L: 0
0109 20:53:06.545159 (+ 171us) write_op.cc:270] Start()
0109 20:53:06.545182 (+ 23us) write_op.cc:276] Timestamp: P: 1736455974536739 usec, L: 0
0109 20:53:06.545321 (+ 139us) write_op.cc:270] Start()
0109 20:53:06.545341 (+ 20us) write_op.cc:276] Timestamp: P: 1736455974559425 usec, L: 0
0109 20:53:06.545455 (+ 114us) write_op.cc:270] Start()
0109 20:53:06.545491 (+ 36us) write_op.cc:276] Timestamp: P: 1736455974561108 usec, L: 0
0109 20:53:06.545643 (+ 152us) write_op.cc:270] Start()
0109 20:53:06.545665 (+ 22us) write_op.cc:276] Timestamp: P: 1736455974567984 usec, L: 0
0109 20:53:06.545804 (+ 139us) write_op.cc:270] Start()
0109 20:53:06.545826 (+ 22us) write_op.cc:276] Timestamp: P: 1736455974776071 usec, L: 0
0109 20:53:06.545968 (+ 142us) write_op.cc:270] Start()
0109 20:53:06.545985 (+ 17us) write_op.cc:276] Timestamp: P: 1736455974787171 usec, L: 0
0109 20:53:06.546102 (+ 117us) write_op.cc:270] Start()
0109 20:53:06.546120 (+ 18us) write_op.cc:276] Timestamp: P: 1736455974787816 usec, L: 0
0109 20:53:06.546240 (+ 120us) write_op.cc:270] Start()
0109 20:53:06.546257 (+ 17us) write_op.cc:276] Timestamp: P: 1736455974838209 usec, L: 0
0109 20:53:06.546364 (+ 107us) write_op.cc:270] Start()
0109 20:53:06.546382 (+ 18us) write_op.cc:276] Timestamp: P: 1736455974839844 usec, L: 0
0109 20:53:06.546502 (+ 120us) write_op.cc:270] Start()
0109 20:53:06.546520 (+ 18us) write_op.cc:276] Timestamp: P: 1736455974843841 usec, L: 0
0109 20:53:06.546661 (+ 141us) write_op.cc:270] Start()
0109 20:53:06.546691 (+ 30us) write_op.cc:276] Timestamp: P: 1736455974844509 usec, L: 0
0109 20:53:06.546817 (+ 126us) write_op.cc:270] Start()
0109 20:53:06.546837 (+ 20us) write_op.cc:276] Timestamp: P: 1736455974858192 usec, L: 0
0109 20:53:06.546973 (+ 136us) write_op.cc:270] Start()
0109 20:53:06.546990 (+ 17us) write_op.cc:276] Timestamp: P: 1736455974858831 usec, L: 0
0109 20:53:06.550047 (+ 3057us) write_op.cc:270] Start()
0109 20:53:06.550073 (+ 26us) write_op.cc:276] Timestamp: P: 1736455974900453 usec, L: 0
0109 20:53:06.550257 (+ 184us) write_op.cc:270] Start()
0109 20:53:06.550276 (+ 19us) write_op.cc:276] Timestamp: P: 1736455974904981 usec, L: 0
0109 20:53:06.550394 (+ 118us) write_op.cc:270] Start()
0109 20:53:06.550413 (+ 19us) write_op.cc:276] Timestamp: P: 1736455974905618 usec, L: 0
0109 20:53:06.550518 (+ 105us) write_op.cc:270] Start()
0109 20:53:06.550537 (+ 19us) write_op.cc:276] Timestamp: P: 1736455974907402 usec, L: 0
0109 20:53:06.550667 (+ 130us) write_op.cc:270] Start()
0109 20:53:06.550694 (+ 27us) write_op.cc:276] Timestamp: P: 1736455974908472 usec, L: 0
0109 20:53:06.550810 (+ 116us) write_op.cc:270] Start()
0109 20:53:06.550828 (+ 18us) write_op.cc:276] Timestamp: P: 1736455974912466 usec, L: 0
0109 20:53:06.551047 (+ 219us) write_op.cc:270] Start()
0109 20:53:06.551072 (+ 25us) write_op.cc:276] Timestamp: P: 1736455974913010 usec, L: 0
0109 20:53:06.553790 (+ 2718us) write_op.cc:270] Start()
0109 20:53:06.553815 (+ 25us) write_op.cc:276] Timestamp: P: 1736455974987664 usec, L: 0
0109 20:53:06.553971 (+ 156us) write_op.cc:270] Start()
0109 20:53:06.553991 (+ 20us) write_op.cc:276] Timestamp: P: 1736455974988233 usec, L: 0
0109 20:53:06.554100 (+ 109us) write_op.cc:270] Start()
0109 20:53:06.554126 (+ 26us) write_op.cc:276] Timestamp: P: 1736455974988804 usec, L: 0
0109 20:53:06.554233 (+ 107us) write_op.cc:270] Start()
0109 20:53:06.554252 (+ 19us) write_op.cc:276] Timestamp: P: 1736455975008116 usec, L: 0
0109 20:53:06.554338 (+ 86us) write_op.cc:270] Start()
0109 20:53:06.554352 (+ 14us) write_op.cc:276] Timestamp: P: 1736455975009375 usec, L: 0
0109 20:53:06.554437 (+ 85us) write_op.cc:270] Start()
0109 20:53:06.554450 (+ 13us) write_op.cc:276] Timestamp: P: 1736455975010216 usec, L: 0
0109 20:53:06.554530 (+ 80us) write_op.cc:270] Start()
0109 20:53:06.554542 (+ 12us) write_op.cc:276] Timestamp: P: 1736455975016046 usec, L: 0
0109 20:53:06.554689 (+ 147us) write_op.cc:270] Start()
0109 20:53:06.554712 (+ 23us) write_op.cc:276] Timestamp: P: 1736455975040744 usec, L: 0
0109 20:53:06.554935 (+ 223us) write_op.cc:270] Start()
0109 20:53:06.554960 (+ 25us) write_op.cc:276] Timestamp: P: 1736455975056806 usec, L: 0
0109 20:53:06.555097 (+ 137us) write_op.cc:270] Start()
0109 20:53:06.555117 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975057385 usec, L: 0
0109 20:53:06.555265 (+ 148us) write_op.cc:270] Start()
0109 20:53:06.555286 (+ 21us) write_op.cc:276] Timestamp: P: 1736455975067121 usec, L: 0
0109 20:53:06.555443 (+ 157us) write_op.cc:270] Start()
0109 20:53:06.555483 (+ 40us) write_op.cc:276] Timestamp: P: 1736455975067638 usec, L: 0
0109 20:53:06.555680 (+ 197us) write_op.cc:270] Start()
0109 20:53:06.555703 (+ 23us) write_op.cc:276] Timestamp: P: 1736455975077232 usec, L: 0
0109 20:53:06.555855 (+ 152us) write_op.cc:270] Start()
0109 20:53:06.555889 (+ 34us) write_op.cc:276] Timestamp: P: 1736455975077718 usec, L: 0
0109 20:53:06.556041 (+ 152us) write_op.cc:270] Start()
0109 20:53:06.556100 (+ 59us) write_op.cc:276] Timestamp: P: 1736455975082183 usec, L: 0
0109 20:53:06.556237 (+ 137us) write_op.cc:270] Start()
0109 20:53:06.556264 (+ 27us) write_op.cc:276] Timestamp: P: 1736455975106652 usec, L: 0
0109 20:53:06.558722 (+ 2458us) write_op.cc:270] Start()
0109 20:53:06.558753 (+ 31us) write_op.cc:276] Timestamp: P: 1736455975111994 usec, L: 0
0109 20:53:06.558934 (+ 181us) write_op.cc:270] Start()
0109 20:53:06.558954 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975124319 usec, L: 0
0109 20:53:06.559074 (+ 120us) write_op.cc:270] Start()
0109 20:53:06.559096 (+ 22us) write_op.cc:276] Timestamp: P: 1736455975124890 usec, L: 0
0109 20:53:06.565036 (+ 5940us) write_op.cc:270] Start()
0109 20:53:06.565066 (+ 30us) write_op.cc:276] Timestamp: P: 1736455975125413 usec, L: 0
0109 20:53:06.565234 (+ 168us) write_op.cc:270] Start()
0109 20:53:06.565254 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975136849 usec, L: 0
0109 20:53:06.565449 (+ 195us) write_op.cc:270] Start()
0109 20:53:06.565488 (+ 39us) write_op.cc:276] Timestamp: P: 1736455975187066 usec, L: 0
0109 20:53:06.565649 (+ 161us) write_op.cc:270] Start()
0109 20:53:06.565676 (+ 27us) write_op.cc:276] Timestamp: P: 1736455975192513 usec, L: 0
0109 20:53:06.565854 (+ 178us) write_op.cc:270] Start()
0109 20:53:06.565885 (+ 31us) write_op.cc:276] Timestamp: P: 1736455975193080 usec, L: 0
0109 20:53:06.566018 (+ 133us) write_op.cc:270] Start()
0109 20:53:06.566036 (+ 18us) write_op.cc:276] Timestamp: P: 1736455975236147 usec, L: 0
0109 20:53:06.566158 (+ 122us) write_op.cc:270] Start()
0109 20:53:06.566177 (+ 19us) write_op.cc:276] Timestamp: P: 1736455975238680 usec, L: 0
0109 20:53:06.566329 (+ 152us) write_op.cc:270] Start()
0109 20:53:06.566349 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975244296 usec, L: 0
0109 20:53:06.566465 (+ 116us) write_op.cc:270] Start()
0109 20:53:06.566483 (+ 18us) write_op.cc:276] Timestamp: P: 1736455975244913 usec, L: 0
0109 20:53:06.566592 (+ 109us) write_op.cc:270] Start()
0109 20:53:06.566629 (+ 37us) write_op.cc:276] Timestamp: P: 1736455975251091 usec, L: 0
0109 20:53:06.566752 (+ 123us) write_op.cc:270] Start()
0109 20:53:06.566768 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975251576 usec, L: 0
0109 20:53:06.566905 (+ 137us) write_op.cc:270] Start()
0109 20:53:06.566929 (+ 24us) write_op.cc:276] Timestamp: P: 1736455975253249 usec, L: 0
0109 20:53:06.567072 (+ 143us) write_op.cc:270] Start()
0109 20:53:06.567090 (+ 18us) write_op.cc:276] Timestamp: P: 1736455975274695 usec, L: 0
0109 20:53:06.567206 (+ 116us) write_op.cc:270] Start()
0109 20:53:06.567223 (+ 17us) write_op.cc:276] Timestamp: P: 1736455975284229 usec, L: 0
0109 20:53:06.567325 (+ 102us) write_op.cc:270] Start()
0109 20:53:06.567340 (+ 15us) write_op.cc:276] Timestamp: P: 1736455975305217 usec, L: 0
0109 20:53:06.567505 (+ 165us) write_op.cc:270] Start()
0109 20:53:06.567524 (+ 19us) write_op.cc:276] Timestamp: P: 1736455975305839 usec, L: 0
0109 20:53:06.567679 (+ 155us) write_op.cc:270] Start()
0109 20:53:06.567700 (+ 21us) write_op.cc:276] Timestamp: P: 1736455975309268 usec, L: 0
0109 20:53:06.567845 (+ 145us) write_op.cc:270] Start()
0109 20:53:06.567860 (+ 15us) write_op.cc:276] Timestamp: P: 1736455975315215 usec, L: 0
0109 20:53:06.567981 (+ 121us) write_op.cc:270] Start()
0109 20:53:06.567998 (+ 17us) write_op.cc:276] Timestamp: P: 1736455975318377 usec, L: 0
0109 20:53:06.568149 (+ 151us) write_op.cc:270] Start()
0109 20:53:06.568172 (+ 23us) write_op.cc:276] Timestamp: P: 1736455975340053 usec, L: 0
0109 20:53:06.568329 (+ 157us) write_op.cc:270] Start()
0109 20:53:06.568351 (+ 22us) write_op.cc:276] Timestamp: P: 1736455975351412 usec, L: 0
0109 20:53:06.568488 (+ 137us) write_op.cc:270] Start()
0109 20:53:06.568504 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975353733 usec, L: 0
0109 20:53:06.607987 (+ 39483us) write_op.cc:270] Start()
0109 20:53:06.608026 (+ 39us) write_op.cc:276] Timestamp: P: 1736455975358267 usec, L: 0
0109 20:53:06.608251 (+ 225us) write_op.cc:270] Start()
0109 20:53:06.608277 (+ 26us) write_op.cc:276] Timestamp: P: 1736455975360174 usec, L: 0
0109 20:53:06.608413 (+ 136us) write_op.cc:270] Start()
0109 20:53:06.608430 (+ 17us) write_op.cc:276] Timestamp: P: 1736455975403323 usec, L: 0
0109 20:53:06.608542 (+ 112us) write_op.cc:270] Start()
0109 20:53:06.608560 (+ 18us) write_op.cc:276] Timestamp: P: 1736455975408679 usec, L: 0
0109 20:53:06.608673 (+ 113us) write_op.cc:270] Start()
0109 20:53:06.608745 (+ 72us) write_op.cc:276] Timestamp: P: 1736455975415737 usec, L: 0
0109 20:53:06.620900 (+ 12155us) write_op.cc:270] Start()
0109 20:53:06.620929 (+ 29us) write_op.cc:276] Timestamp: P: 1736455975419759 usec, L: 0
0109 20:53:06.621105 (+ 176us) write_op.cc:270] Start()
0109 20:53:06.621124 (+ 19us) write_op.cc:276] Timestamp: P: 1736455975450068 usec, L: 0
0109 20:53:06.621247 (+ 123us) write_op.cc:270] Start()
0109 20:53:06.621264 (+ 17us) write_op.cc:276] Timestamp: P: 1736455975458784 usec, L: 0
0109 20:53:06.621361 (+ 97us) write_op.cc:270] Start()
0109 20:53:06.621377 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975470101 usec, L: 0
0109 20:53:06.621492 (+ 115us) write_op.cc:270] Start()
0109 20:53:06.621572 (+ 80us) write_op.cc:276] Timestamp: P: 1736455975477560 usec, L: 0
0109 20:53:06.621726 (+ 154us) write_op.cc:270] Start()
0109 20:53:06.621746 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975478091 usec, L: 0
0109 20:53:06.621848 (+ 102us) write_op.cc:270] Start()
0109 20:53:06.621863 (+ 15us) write_op.cc:276] Timestamp: P: 1736455975482457 usec, L: 0
0109 20:53:06.621993 (+ 130us) write_op.cc:270] Start()
0109 20:53:06.622022 (+ 29us) write_op.cc:276] Timestamp: P: 1736455975484478 usec, L: 0
0109 20:53:06.622159 (+ 137us) write_op.cc:270] Start()
0109 20:53:06.622177 (+ 18us) write_op.cc:276] Timestamp: P: 1736455975503512 usec, L: 0
0109 20:53:06.622295 (+ 118us) write_op.cc:270] Start()
0109 20:53:06.622311 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975513505 usec, L: 0
0109 20:53:06.622419 (+ 108us) write_op.cc:270] Start()
0109 20:53:06.622435 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975522850 usec, L: 0
0109 20:53:06.622547 (+ 112us) write_op.cc:270] Start()
0109 20:53:06.622563 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975541351 usec, L: 0
0109 20:53:06.622673 (+ 110us) write_op.cc:270] Start()
0109 20:53:06.622689 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975541939 usec, L: 0
0109 20:53:06.622814 (+ 125us) write_op.cc:270] Start()
0109 20:53:06.622834 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975546256 usec, L: 0
0109 20:53:06.663023 (+ 40189us) write_op.cc:270] Start()
0109 20:53:06.663056 (+ 33us) write_op.cc:276] Timestamp: P: 1736455975546693 usec, L: 0
0109 20:53:06.663233 (+ 177us) write_op.cc:270] Start()
0109 20:53:06.663255 (+ 22us) write_op.cc:276] Timestamp: P: 1736455975564258 usec, L: 0
0109 20:53:06.663389 (+ 134us) write_op.cc:270] Start()
0109 20:53:06.663406 (+ 17us) write_op.cc:276] Timestamp: P: 1736455975566259 usec, L: 0
0109 20:53:06.663533 (+ 127us) write_op.cc:270] Start()
0109 20:53:06.663553 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975579184 usec, L: 0
0109 20:53:06.663703 (+ 150us) write_op.cc:270] Start()
0109 20:53:06.663723 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975606798 usec, L: 0
0109 20:53:06.663853 (+ 130us) write_op.cc:270] Start()
0109 20:53:06.663883 (+ 30us) write_op.cc:276] Timestamp: P: 1736455975607348 usec, L: 0
0109 20:53:06.664017 (+ 134us) write_op.cc:270] Start()
0109 20:53:06.664037 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975628629 usec, L: 0
0109 20:53:06.664198 (+ 161us) write_op.cc:270] Start()
0109 20:53:06.664218 (+ 20us) write_op.cc:276] Timestamp: P: 1736455975629356 usec, L: 0
0109 20:53:06.664344 (+ 126us) write_op.cc:270] Start()
0109 20:53:06.664374 (+ 30us) write_op.cc:276] Timestamp: P: 1736455975664051 usec, L: 0
0109 20:53:06.664492 (+ 118us) write_op.cc:270] Start()
0109 20:53:06.664509 (+ 17us) write_op.cc:276] Timestamp: P: 1736455975686925 usec, L: 0
0109 20:53:06.664637 (+ 128us) write_op.cc:270] Start()
0109 20:53:06.664655 (+ 18us) write_op.cc:276] Timestamp: P: 1736455975690511 usec, L: 0
0109 20:53:06.664860 (+ 205us) write_op.cc:270] Start()
0109 20:53:06.664889 (+ 29us) write_op.cc:276] Timestamp: P: 1736455975693406 usec, L: 0
0109 20:53:06.665024 (+ 135us) write_op.cc:270] Start()
0109 20:53:06.665040 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975697094 usec, L: 0
0109 20:53:06.665153 (+ 113us) write_op.cc:270] Start()
0109 20:53:06.665170 (+ 17us) write_op.cc:276] Timestamp: P: 1736455975701727 usec, L: 0
0109 20:53:06.665294 (+ 124us) write_op.cc:270] Start()
0109 20:53:06.665312 (+ 18us) write_op.cc:276] Timestamp: P: 1736455975703052 usec, L: 0
0109 20:53:06.665429 (+ 117us) write_op.cc:270] Start()
0109 20:53:06.665445 (+ 16us) write_op.cc:276] Timestamp: P: 1736455975707122 usec, L: 0
0109 20:53:06.665587 (+ 142us) write_op.cc:270] Start()
0109 20:53:06.665609 (+ 22us) write_op.cc:276] Timestamp: P: 1736455975712960 usec, L: 0
0109 20:53:06.665731 (+ 122us) write_op.cc:270] Start()
0109 20:53:06.665748 (+ 17us) write_op.cc:276] Timestamp: P: 1736455975743635 usec, L: 0
0109 20:53:06.665898 (+ 150us) write_op.cc:270] Start()
0109 20:53:06.665919 (+ 21us) write_op.cc:276] Timestamp: P: 1736455975768495 usec, L: 0
0109 20:53:06.666065 (+ 146us) write_op.cc:270] Start()
0109 20:53:06.666088 (+ 23us) write_op.cc:276] Timestamp: P: 1736455975774468 usec, L: 0
0109 20:53:06.666234 (+ 146us) write_op.cc:270] Start()
0109 20:53:06.666254 (+ 20us) write_op.cc:276] Timestamp: P: 1736455
I20250109 20:53:11.210433 1158 tablet_bootstrap.cc:492] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.210739 1158 ts_tablet_manager.cc:1397] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.003s
I20250109 20:53:11.200060 2802 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } }
I20250109 20:53:11.211884 2760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:53:11.212070 2760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: LEADER
I20250109 20:53:11.212677 1158 raft_consensus.cc:357] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.213265 1158 raft_consensus.cc:383] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.213460 1158 raft_consensus.cc:738] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.213899 1158 consensus_queue.cc:260] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.215306 1158 ts_tablet_manager.cc:1428] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:11.215927 1158 tablet_bootstrap.cc:492] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.222172 2749 raft_consensus.cc:491] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.222282 2749 raft_consensus.cc:513] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.222730 2749 leader_election.cc:290] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.223157 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc803f2936b4538a5e70ef34e5cc5d4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:11.223832 990 leader_election.cc:343] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.224200 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc803f2936b4538a5e70ef34e5cc5d4" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:11.224215 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1210 } } ignore_live_leader: false dest_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" is_pre_election: true
W20250109 20:53:11.224591 989 leader_election.cc:343] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.224673 989 leader_election.cc:304] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.225319 2749 raft_consensus.cc:2743] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.228499 1165 tablet_bootstrap.cc:492] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.231531 1165 ts_tablet_manager.cc:1397] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20250109 20:53:11.232251 1158 tablet_bootstrap.cc:654] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.233434 1176 tablet_bootstrap.cc:492] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.235898 1176 ts_tablet_manager.cc:1397] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.007s
I20250109 20:53:11.245038 2818 raft_consensus.cc:491] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.245172 2818 raft_consensus.cc:513] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.246184 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:11.247314 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85c36689d8f4b16b85db5ad3c84cac0" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:11.247056 2818 leader_election.cc:290] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:11.247630 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 343562us
I20250109 20:53:11.247871 854 leader_election.cc:304] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.248086 1157 raft_consensus.cc:2267] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.248348 713 leader_election.cc:304] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.248641 2818 raft_consensus.cc:2743] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.248867 2754 raft_consensus.cc:2798] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:11.248972 2754 raft_consensus.cc:695] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:11.249181 2754 consensus_queue.cc:237] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.249454 713 leader_election.cc:304] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.236833 1176 raft_consensus.cc:357] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.249718 1176 raft_consensus.cc:383] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.249774 1176 raft_consensus.cc:738] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.249871 1157 ts_tablet_manager.cc:1905] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.251065 2802 leader_election.cc:290] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8328960890b142248073cec450e957d8 (127.0.104.62:37941), 0f066b28f8dd4e9cae5c64bd93feeb51 (127.0.104.61:39231)
I20250109 20:53:11.251394 2695 raft_consensus.cc:2798] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:11.251482 2695 raft_consensus.cc:491] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:11.251552 2695 raft_consensus.cc:3054] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:11.246865 1165 raft_consensus.cc:357] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.260182 1165 raft_consensus.cc:383] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.260833 1165 raft_consensus.cc:738] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.263168 1157 ts_tablet_manager.cc:1918] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:11.263262 1157 log.cc:1198] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/70b48900f57f47128e31b27275b1489c
I20250109 20:53:11.263679 1157 ts_tablet_manager.cc:1939] T 70b48900f57f47128e31b27275b1489c P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:11.264089 2695 raft_consensus.cc:513] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.264672 1176 consensus_queue.cc:260] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.264992 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b4c038573a4afa879ecbcba75c35a7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:11.265460 712 leader_election.cc:343] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.265646 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]) successfully deleted
W20250109 20:53:11.265862 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet e53966daf1ed4a67801d9ab239e17d8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e53966daf1ed4a67801d9ab239e17d8e already in progress: creating tablet
W20250109 20:53:11.266057 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2298004317b14932bf80f39568ad73da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2298004317b14932bf80f39568ad73da already in progress: creating tablet
W20250109 20:53:11.266332 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 63a665fcf6ba43f59de8b945c5534d8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63a665fcf6ba43f59de8b945c5534d8e already in progress: creating tablet
I20250109 20:53:11.267078 2695 leader_election.cc:290] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.265246 1165 consensus_queue.cc:260] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:11.266732 713 leader_election.cc:343] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 0a2efcc2a6ec4b109845d8087491ca9f
I20250109 20:53:11.268184 713 leader_election.cc:304] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.269659 2826 raft_consensus.cc:491] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.269783 2826 raft_consensus.cc:513] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.269939 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36a29ab2002d4eb389cd11778988e1e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1210 } } ignore_live_leader: false dest_uuid: "8328960890b142248073cec450e957d8" is_pre_election: true
I20250109 20:53:11.270314 2826 leader_election.cc:290] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.270483 2103 leader_election.cc:304] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36a29ab2002d4eb389cd11778988e1e9; no voters: 0f066b28f8dd4e9cae5c64bd93feeb51, 8328960890b142248073cec450e957d8
W20250109 20:53:11.270854 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc54e97e91c54afea098624bd85fcc6e consensus state has no leader
I20250109 20:53:11.270836 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b285905342b3820f9549c8f0c4ad" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:11.271248 990 leader_election.cc:343] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.271652 1176 ts_tablet_manager.cc:1428] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20250109 20:53:11.271829 2807 raft_consensus.cc:2743] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.273562 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 76e688003f5b4a28adfcba0a4ce4b9da
I20250109 20:53:11.273422 449 catalog_manager.cc:5526] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:11.273784 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet ee208bad76734a8fa1ada438bdbbe48c (table default.Table_79fb13be9f0440c9991b67d641ab0b6a [id=238aa3a3a8e74dfba23e60c7d6f049cf]). Expected version 1 got 0
I20250109 20:53:11.273941 449 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cc54e97e91c54afea098624bd85fcc6e (table default.Table_bc4981434b6743fc93063e84ec2ed645 [id=28723b313bd34951826646276c761571]). Expected version 3 got 0
I20250109 20:53:11.274132 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c9df47cadc104f3f915d6e77b01d9e1f" } modify_external_catalogs: true
I20250109 20:53:11.274248 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_c9df47cadc104f3f915d6e77b01d9e1f" } modify_external_catalogs: true
I20250109 20:53:11.274811 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b545e21699b5465eb751e33e57a1fe88" } modify_external_catalogs: true
I20250109 20:53:11.274942 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b545e21699b5465eb751e33e57a1fe88" } modify_external_catalogs: true
I20250109 20:53:11.275466 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_229ce97b77fe4d589363707c89b17ef7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.275930 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_229ce97b77fe4d589363707c89b17ef7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.276499 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e688003f5b4a28adfcba0a4ce4b9da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.273427 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_99e81a65fa2741ac80eccc081148192a" } modify_external_catalogs: true
I20250109 20:53:11.277379 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_99e81a65fa2741ac80eccc081148192a" } modify_external_catalogs: true
W20250109 20:53:11.278311 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 046ba213bde34e35b144d3177c62d79f consensus state has no leader
I20250109 20:53:11.278553 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3ebf8106cfdb42bdbf56ec609f15305d" } modify_external_catalogs: true
I20250109 20:53:11.278664 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_3ebf8106cfdb42bdbf56ec609f15305d" } modify_external_catalogs: true
I20250109 20:53:11.280151 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b285905342b3820f9549c8f0c4ad" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.280629 989 leader_election.cc:343] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.280742 989 leader_election.cc:304] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.280970 2826 raft_consensus.cc:2743] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:11.282742 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9615f0d1bb9d46ba8e855fb5819eb5a6 consensus state has no leader
I20250109 20:53:11.283255 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec1d30b67674facb8b92a2c1a408073 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.283083 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_8f3c8dbeed7645a5849426fa972200c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:11.283445 1160 tablet_replica.cc:331] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.283583 1160 raft_consensus.cc:2238] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:11.283612 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f3c8dbeed7645a5849426fa972200c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.283707 1160 raft_consensus.cc:2267] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.284435 1160 ts_tablet_manager.cc:1905] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:11.284621 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7204d58f2cf640059ad827e11f62912a consensus state has no leader
I20250109 20:53:11.284790 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 712acab92e0f44a3a585d7d871498aa3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e020eeb03e041ed977779a07de5bb80) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.286092 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:11.287307 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.284912 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b8451f734d454d20bfa0173a6a0ea0dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.287789 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8451f734d454d20bfa0173a6a0ea0dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.288166 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_6888260a7ce740ba8e754d53fed1e700" } new_table_name: "default.Table_f511e92e0df04dcdad9a77426873b158" modify_external_catalogs: true
I20250109 20:53:11.289469 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5ba80d2ae9b14d0b94259012c5686f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.289844 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ba80d2ae9b14d0b94259012c5686f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.290051 2826 raft_consensus.cc:491] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.290163 2826 raft_consensus.cc:513] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.290948 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fe30752f49478afad9b93dc235eb" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:11.291363 1160 ts_tablet_manager.cc:1918] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.291483 1160 log.cc:1198] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/0ec1d30b67674facb8b92a2c1a408073
W20250109 20:53:11.291541 990 leader_election.cc:343] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.291862 1160 ts_tablet_manager.cc:1939] T 0ec1d30b67674facb8b92a2c1a408073 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:11.291934 2826 leader_election.cc:290] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.292902 1160 tablet_replica.cc:331] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.292994 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fe30752f49478afad9b93dc235eb" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.293370 989 leader_election.cc:343] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.293462 989 leader_election.cc:304] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.294317 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 0ec1d30b67674facb8b92a2c1a408073 (table default.Table_0cbc3b8995704eaf99466fd1d5d0e124 [id=22ffbc424e6d4038a66ddea1fca9842e]) successfully deleted
I20250109 20:53:11.283255 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4abc19297e4029b1b7c5396b49b967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.296952 2826 raft_consensus.cc:2743] T 9629fe30752f49478afad9b93dc235eb P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:11.297942 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5056ac74d94e424aa2e7a79a8c9edd86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5056ac74d94e424aa2e7a79a8c9edd86 already in progress: creating tablet
I20250109 20:53:11.302028 2802 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 36a29ab2002d4eb389cd11778988e1e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250109 20:53:11.305939 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e688003f5b4a28adfcba0a4ce4b9da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:11.306389 713 leader_election.cc:343] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: bdcb6780422541c89343e98e02938d0e
I20250109 20:53:11.306528 713 leader_election.cc:304] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.306787 2807 raft_consensus.cc:2743] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.307150 1165 ts_tablet_manager.cc:1428] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.074s user 0.004s sys 0.000s
I20250109 20:53:11.308033 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 006475691733470f8a0468365972dfd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.308146 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e688003f5b4a28adfcba0a4ce4b9da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.308647 1165 tablet_bootstrap.cc:492] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:53:11.309056 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 76e688003f5b4a28adfcba0a4ce4b9da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e688003f5b4a28adfcba0a4ce4b9da already in progress: creating tablet
W20250109 20:53:11.310087 713 leader_election.cc:343] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2797e1df3d12476f89fb5155523c3935
I20250109 20:53:11.310185 713 leader_election.cc:304] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.310390 2807 raft_consensus.cc:2743] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.311928 1165 tablet_bootstrap.cc:654] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:11.312754 713 leader_election.cc:343] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 3293f49190684b72a5c41031e621bd36
I20250109 20:53:11.312847 713 leader_election.cc:304] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.313165 2695 raft_consensus.cc:2743] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:11.313241 713 leader_election.cc:343] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 3af9563f6efd47059caac7c86c41d43f
I20250109 20:53:11.313313 713 leader_election.cc:304] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.313532 2695 raft_consensus.cc:2743] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.316285 2826 raft_consensus.cc:491] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.316418 2826 raft_consensus.cc:513] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.317333 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb9b0f722eb44b0acd5cb3bbb32c527" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:11.317395 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb9b0f722eb44b0acd5cb3bbb32c527" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.317783 990 leader_election.cc:343] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:11.317906 989 leader_election.cc:343] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.318011 989 leader_election.cc:304] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.318233 2822 raft_consensus.cc:2743] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.318387 2826 leader_election.cc:290] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.319758 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b4c038573a4afa879ecbcba75c35a7" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:11.319916 1051 raft_consensus.cc:3054] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:11.321271 1166 tablet_replica.cc:331] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.321435 1166 raft_consensus.cc:2238] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.321529 1166 raft_consensus.cc:2267] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.321919 1166 ts_tablet_manager.cc:1905] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.322170 1051 raft_consensus.cc:2463] T f7b4c038573a4afa879ecbcba75c35a7 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:11.323731 1158 tablet_bootstrap.cc:492] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.323828 1158 ts_tablet_manager.cc:1397] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.108s user 0.001s sys 0.005s
I20250109 20:53:11.324512 1158 raft_consensus.cc:357] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.324656 1158 raft_consensus.cc:383] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.324692 1158 raft_consensus.cc:738] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.325270 1176 tablet_bootstrap.cc:492] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.325668 1158 consensus_queue.cc:260] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.326149 1158 ts_tablet_manager.cc:1428] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:11.326385 1158 tablet_bootstrap.cc:492] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.326727 1176 tablet_bootstrap.cc:654] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.327873 1158 tablet_bootstrap.cc:654] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.328294 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 39a8898aaba349f2bb111401e63da281 (DEFAULT_TABLE table=default.Table_6ee7a6b6b6c142ac8a14a4fe60c7c5b6 [id=3de085680e1846fca47b98d03384708a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.328922 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a8898aaba349f2bb111401e63da281. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.329028 713 leader_election.cc:304] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.329763 2692 raft_consensus.cc:2798] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:11.329869 2692 raft_consensus.cc:695] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:11.330166 1165 tablet_bootstrap.cc:492] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.330085 2692 consensus_queue.cc:237] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.330276 1165 ts_tablet_manager.cc:1397] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250109 20:53:11.330277 1166 ts_tablet_manager.cc:1918] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.330384 1166 log.cc:1198] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/189cd8079a9d4ca6b1703f4361d5ec5d
I20250109 20:53:11.330894 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 70b48900f57f47128e31b27275b1489c (table default.Table_f53efacb818d49bea258bd79021056d3 [id=b9ba88e68399420c8bffa0407b369aed]) successfully deleted
I20250109 20:53:11.330868 1165 raft_consensus.cc:357] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.331038 1165 raft_consensus.cc:383] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.331081 1165 raft_consensus.cc:738] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.331277 1165 consensus_queue.cc:260] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.332623 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5f55dd45ad16489cbfaa59338a2fbb44
I20250109 20:53:11.335016 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 8e67aa0fafa44a75bceeaff2f05298bc from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:11.336405 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f55dd45ad16489cbfaa59338a2fbb44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.337141 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2efcc2a6ec4b109845d8087491ca9f with delete_type TABLET_DATA_DELETED (replaced by tablet 3fc68008840d4b8fae5185718ab454fa) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:11.336973 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5f55dd45ad16489cbfaa59338a2fbb44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f55dd45ad16489cbfaa59338a2fbb44 already in progress: creating tablet
I20250109 20:53:11.337388 1157 tablet_replica.cc:331] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:11.337568 752 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcb6780422541c89343e98e02938d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.337777 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 6461f153a5994412ba05c3b687963fd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.338515 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f55dd45ad16489cbfaa59338a2fbb44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.342623 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet edc803f2936b4538a5e70ef34e5cc5d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.342818 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 40546e1b5e854a79a7f706a43b6c6062 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:07 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.343413 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet cbef1d27c5394fc1b6354c5d6f3a8f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.337152 739 tablet_service.cc:1514] Processing DeleteTablet for tablet face19003d0042669e3f22553f9fe19a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:55 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.337574 1157 raft_consensus.cc:2238] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.344753 1157 raft_consensus.cc:2267] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.346968 1157 ts_tablet_manager.cc:1905] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.349728 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797e1df3d12476f89fb5155523c3935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.351147 1165 ts_tablet_manager.cc:1428] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250109 20:53:11.359345 2788 tablet.cc:1721] T d845bac2539e478083bf7fe6cbc0c717 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:11.361166 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f55dd45ad16489cbfaa59338a2fbb44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.363067 2754 consensus_queue.cc:1035] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:11.363258 2835 tablet.cc:1721] T d845bac2539e478083bf7fe6cbc0c717 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250109 20:53:11.364590 1158 tablet_bootstrap.cc:492] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.364703 1158 ts_tablet_manager.cc:1397] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20250109 20:53:11.366526 1158 raft_consensus.cc:357] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.366704 1158 raft_consensus.cc:383] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.366752 1158 raft_consensus.cc:738] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.366930 1158 consensus_queue.cc:260] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.367458 1158 ts_tablet_manager.cc:1428] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250109 20:53:11.367735 1158 tablet_bootstrap.cc:492] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.370213 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_28232db1d8254b5fa6eb238dcb283d36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.370882 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28232db1d8254b5fa6eb238dcb283d36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.371616 1176 tablet_bootstrap.cc:492] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.371745 1176 ts_tablet_manager.cc:1397] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.003s
I20250109 20:53:11.372567 1176 raft_consensus.cc:357] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.372823 1176 raft_consensus.cc:383] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.372884 1176 raft_consensus.cc:738] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.373104 1176 consensus_queue.cc:260] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.373805 1176 ts_tablet_manager.cc:1428] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250109 20:53:11.374107 1176 tablet_bootstrap.cc:492] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.375771 1176 tablet_bootstrap.cc:654] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.376235 1158 tablet_bootstrap.cc:654] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:11.376966 712 consensus_peers.cc:487] T ee208bad76734a8fa1ada438bdbbe48c P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:11.383558 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 5bfa04c33e104c148a73c545284c20a1
I20250109 20:53:11.384634 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfa04c33e104c148a73c545284c20a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.384861 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfa04c33e104c148a73c545284c20a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.386338 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfa04c33e104c148a73c545284c20a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.386581 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 05df026727da4ea594810bb9b516dc7c
I20250109 20:53:11.387565 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 05df026727da4ea594810bb9b516dc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.387624 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 05df026727da4ea594810bb9b516dc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.387768 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 05df026727da4ea594810bb9b516dc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.390064 2838 tablet.cc:1721] T d845bac2539e478083bf7fe6cbc0c717 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250109 20:53:11.393604 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7204d58f2cf640059ad827e11f62912a (table default.Table_b1a1839ad80842dda6e5ed8a290f89ab [id=4ad71ffbd3b5478087f0cd6b3a3f7bbf], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7204d58f2cf640059ad827e11f62912a consensus state has no leader
I20250109 20:53:11.399643 1160 raft_consensus.cc:2238] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.400117 1160 raft_consensus.cc:2267] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.401268 1176 tablet_bootstrap.cc:492] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.401479 1176 ts_tablet_manager.cc:1397] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20250109 20:53:11.402204 1176 raft_consensus.cc:357] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.402426 1176 raft_consensus.cc:383] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.402504 1176 raft_consensus.cc:738] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.402730 1176 consensus_queue.cc:260] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.403455 1176 ts_tablet_manager.cc:1428] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:11.405591 1176 tablet_bootstrap.cc:492] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.405609 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 88479a3f57654d84ad622af0782cce47 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:11.409667 1160 ts_tablet_manager.cc:1905] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.411453 1165 tablet_bootstrap.cc:492] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.412951 1165 tablet_bootstrap.cc:654] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.413187 2826 raft_consensus.cc:491] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.413295 2826 raft_consensus.cc:513] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.413793 2826 leader_election.cc:290] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:11.414352 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eaa5bbfaf4c1c9d8cd1b1ead5f632" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:11.414940 990 leader_election.cc:343] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.415324 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eaa5bbfaf4c1c9d8cd1b1ead5f632" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.415818 989 leader_election.cc:343] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.415933 989 leader_election.cc:304] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.416325 2826 raft_consensus.cc:2743] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.418900 1157 ts_tablet_manager.cc:1918] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.431803 1176 tablet_bootstrap.cc:654] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:11.433110 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu pid 1922: Timed out: CheckLeaks RPC to 127.0.104.60:41583 timed out after 1.000s (SENT)
I20250109 20:53:11.433245 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu with pid 1922
I20250109 20:53:11.434425 1157 log.cc:1198] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/0a2efcc2a6ec4b109845d8087491ca9f
I20250109 20:53:11.435034 1157 ts_tablet_manager.cc:1939] T 0a2efcc2a6ec4b109845d8087491ca9f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:11.441350 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 2b3af68b009a4789bc1cdbafc8d44dcf
I20250109 20:53:11.442420 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3af68b009a4789bc1cdbafc8d44dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:11.445335 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a279331730c7482580650c24365c4cba consensus state has no leader
I20250109 20:53:11.446043 1160 ts_tablet_manager.cc:1918] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.446139 1160 log.cc:1198] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/712acab92e0f44a3a585d7d871498aa3
I20250109 20:53:11.449496 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3af68b009a4789bc1cdbafc8d44dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.450932 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_3f395e4c8c84426aa4888bd424686c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.451527 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f395e4c8c84426aa4888bd424686c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.451557 454 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet ee208bad76734a8fa1ada438bdbbe48c (table default.Table_79fb13be9f0440c9991b67d641ab0b6a [id=238aa3a3a8e74dfba23e60c7d6f049cf]). Expected version 1 got 0
I20250109 20:53:11.452673 1160 ts_tablet_manager.cc:1939] T 712acab92e0f44a3a585d7d871498aa3 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:11.453439 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 712acab92e0f44a3a585d7d871498aa3 (table default.Table_1f4279f07e8b4e0a86928bdb9a67bbf6 [id=8bec14b8b4b74d9c87fe86950f5128e1]) successfully deleted
I20250109 20:53:11.453935 1160 tablet_replica.cc:331] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.454113 1160 raft_consensus.cc:2238] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.454260 1160 raft_consensus.cc:2267] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.454989 1176 tablet_bootstrap.cc:492] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.455107 1176 ts_tablet_manager.cc:1397] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.002s
I20250109 20:53:11.455749 1176 raft_consensus.cc:357] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.455905 1176 raft_consensus.cc:383] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.455947 1176 raft_consensus.cc:738] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.456172 1176 consensus_queue.cc:260] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.456696 1176 ts_tablet_manager.cc:1428] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:11.456987 1176 tablet_bootstrap.cc:492] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.457245 451 catalog_manager.cc:5526] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:11.457927 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet c0860554cac64b8882e39fa62462b1aa (table default.Table_f74ab9f194134e6ba390983fba09d20a [id=f36b9d8c432d4f108296cd5576832573]). Expected version 2 got 0
I20250109 20:53:11.458101 1176 tablet_bootstrap.cc:654] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.458204 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d3681efa8785480ca65a51b4b40448f1 (table default.Table_e6c36b2fd8a749f296aeca3e9191d0f8 [id=6e40bdc674a9444181bd7deacfdea27d]). Expected version 1 got 0
I20250109 20:53:11.458567 451 catalog_manager.cc:5526] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:11.459017 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet cc54e97e91c54afea098624bd85fcc6e (table default.Table_bc4981434b6743fc93063e84ec2ed645 [id=28723b313bd34951826646276c761571]). Expected version 3 got 0
I20250109 20:53:11.459267 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:11.461125 451 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet ee208bad76734a8fa1ada438bdbbe48c (table default.Table_79fb13be9f0440c9991b67d641ab0b6a [id=238aa3a3a8e74dfba23e60c7d6f049cf]). Expected version 1 got 0
I20250109 20:53:11.463704 1166 ts_tablet_manager.cc:1939] T 189cd8079a9d4ca6b1703f4361d5ec5d P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.466054 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 189cd8079a9d4ca6b1703f4361d5ec5d (table default.Table_7de47568081c4d11a33a67edd6898f8f [id=f4be51ab8d8b4a1a9238e74cb1d3e022]) successfully deleted
I20250109 20:53:11.467273 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 8e67aa0fafa44a75bceeaff2f05298bc
I20250109 20:53:11.467700 449 catalog_manager.cc:6596] ReplaceTablet: tablet 8e67aa0fafa44a75bceeaff2f05298bc deleted and replaced by tablet 8ac0dbb5e1dc41a5b6e50fe38ce97bf0
I20250109 20:53:11.468537 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e67aa0fafa44a75bceeaff2f05298bc with delete_type TABLET_DATA_DELETED (replaced by tablet 8ac0dbb5e1dc41a5b6e50fe38ce97bf0) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.468827 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e67aa0fafa44a75bceeaff2f05298bc with delete_type TABLET_DATA_DELETED (replaced by tablet 8ac0dbb5e1dc41a5b6e50fe38ce97bf0) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:11.469527 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2298004317b14932bf80f39568ad73da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2298004317b14932bf80f39568ad73da already in progress: creating tablet
W20250109 20:53:11.470217 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 63a665fcf6ba43f59de8b945c5534d8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63a665fcf6ba43f59de8b945c5534d8e already in progress: creating tablet
I20250109 20:53:11.442104 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3af68b009a4789bc1cdbafc8d44dcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.470719 1176 tablet_bootstrap.cc:492] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.470840 1176 ts_tablet_manager.cc:1397] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20250109 20:53:11.471386 1176 raft_consensus.cc:357] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.471617 1176 raft_consensus.cc:383] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.471668 1176 raft_consensus.cc:738] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.471836 1176 consensus_queue.cc:260] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.472726 1176 ts_tablet_manager.cc:1428] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:11.472929 1176 tablet_bootstrap.cc:492] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.472987 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 0a2efcc2a6ec4b109845d8087491ca9f (table default.Table_0c702c1df34a466da3d2c24f0150e438 [id=29268647fb064459a12b1d969b9e2c22]) successfully deleted
I20250109 20:53:11.473311 1157 tablet_replica.cc:331] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:11.473522 1157 raft_consensus.cc:2238] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.473711 1157 raft_consensus.cc:2267] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.473935 1160 ts_tablet_manager.cc:1905] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.474181 1157 ts_tablet_manager.cc:1905] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.474392 1176 tablet_bootstrap.cc:654] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.475525 1165 tablet_bootstrap.cc:492] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.475641 1165 ts_tablet_manager.cc:1397] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.004s
I20250109 20:53:11.468196 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e67aa0fafa44a75bceeaff2f05298bc with delete_type TABLET_DATA_DELETED (replaced by tablet 8ac0dbb5e1dc41a5b6e50fe38ce97bf0) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:11.476612 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e53966daf1ed4a67801d9ab239e17d8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e53966daf1ed4a67801d9ab239e17d8e already in progress: creating tablet
I20250109 20:53:11.476997 1166 tablet_replica.cc:331] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.477998 1166 raft_consensus.cc:2238] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.478202 1166 raft_consensus.cc:2267] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.479866 1158 tablet_bootstrap.cc:492] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.479995 1158 ts_tablet_manager.cc:1397] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.112s user 0.002s sys 0.008s
I20250109 20:53:11.480619 1158 raft_consensus.cc:357] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.480796 1158 raft_consensus.cc:383] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.480844 1158 raft_consensus.cc:738] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.481029 1158 consensus_queue.cc:260] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.481475 1158 ts_tablet_manager.cc:1428] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:11.481492 1166 ts_tablet_manager.cc:1905] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.481740 1158 tablet_bootstrap.cc:492] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.487654 1157 ts_tablet_manager.cc:1918] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.488543 1157 log.cc:1198] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/bdcb6780422541c89343e98e02938d0e
I20250109 20:53:11.489091 1157 ts_tablet_manager.cc:1939] T bdcb6780422541c89343e98e02938d0e P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:11.490542 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_2c4291be2f56426cafb273138c51dece"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.491072 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c4291be2f56426cafb273138c51dece in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.492287 1176 tablet_bootstrap.cc:492] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.492411 1176 ts_tablet_manager.cc:1397] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250109 20:53:11.493011 1176 raft_consensus.cc:357] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.493149 449 master_service.cc:946] ReplaceTablet: received request to replace tablet feb9b0f722eb44b0acd5cb3bbb32c527 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:11.493176 1176 raft_consensus.cc:383] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.493230 1176 raft_consensus.cc:738] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.493430 1176 consensus_queue.cc:260] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.493948 1176 ts_tablet_manager.cc:1428] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250109 20:53:11.494158 1176 tablet_bootstrap.cc:492] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.495815 1176 tablet_bootstrap.cc:654] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.496446 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_908fd543e71a4471a5c0f402e7ac5a7a" } modify_external_catalogs: true
I20250109 20:53:11.496616 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_908fd543e71a4471a5c0f402e7ac5a7a" } modify_external_catalogs: true
I20250109 20:53:11.496951 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a2156eacd7fd4fcba8e1649d9a83ad81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.497397 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2156eacd7fd4fcba8e1649d9a83ad81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.497802 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_915d920262ad443daadf80f5f7367d69" } new_table_name: "default.Table_87f3dbf3f44842bfa78405e88b9b411b" modify_external_catalogs: true
I20250109 20:53:11.499219 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d0aaf004b27e42c8903f69a38b65289f" } new_table_name: "default.Table_29cd1a37623848feb9407126657e39fc" modify_external_catalogs: true
I20250109 20:53:11.499656 1166 ts_tablet_manager.cc:1918] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.499737 1166 log.cc:1198] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/feb9b0f722eb44b0acd5cb3bbb32c527
I20250109 20:53:11.499984 1166 ts_tablet_manager.cc:1939] T feb9b0f722eb44b0acd5cb3bbb32c527 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.502743 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet feb9b0f722eb44b0acd5cb3bbb32c527 (table default.Table_bbc5532de8774ffc9448c2680924697f [id=1b732f5cb5904387b841b5be096d09c5]) successfully deleted
W20250109 20:53:11.503264 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5056ac74d94e424aa2e7a79a8c9edd86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5056ac74d94e424aa2e7a79a8c9edd86 already in progress: creating tablet
I20250109 20:53:11.503597 1166 tablet_replica.cc:331] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.503768 1166 raft_consensus.cc:2238] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.503873 1166 raft_consensus.cc:2267] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.504346 1166 ts_tablet_manager.cc:1905] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.508913 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet bdcb6780422541c89343e98e02938d0e (table default.Table_cf2a9e7391e84accaa9cfc8a4f69b3fe [id=fe5e7a3f2800480e860c07f214cfc841]) successfully deleted
I20250109 20:53:11.509207 1157 tablet_replica.cc:331] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:11.509430 1157 raft_consensus.cc:2238] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.513625 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 88479a3f57654d84ad622af0782cce47
I20250109 20:53:11.513983 452 catalog_manager.cc:6596] ReplaceTablet: tablet 88479a3f57654d84ad622af0782cce47 deleted and replaced by tablet 43df4de52ddd4765984aee1c4d2479da
I20250109 20:53:11.514865 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_c8691eae9ae94278acb07e41ff9e1cfe [id=2509b1e547034a9ea53a95d04a14cf0b] alter complete (version 2)
I20250109 20:53:11.523943 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 88479a3f57654d84ad622af0782cce47 with delete_type TABLET_DATA_DELETED (replaced by tablet 43df4de52ddd4765984aee1c4d2479da) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.523973 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e716270ffcf44039b075023fd0c2d2a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.525018 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6ec8f3d4f4d37a11f15173449155c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.525179 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 80faa29a3b314796b68a4073665d2898 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:56 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.525306 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 88479a3f57654d84ad622af0782cce47 with delete_type TABLET_DATA_DELETED (replaced by tablet 43df4de52ddd4765984aee1c4d2479da) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.525733 1166 ts_tablet_manager.cc:1918] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.525815 1166 log.cc:1198] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/f1534c7240fc49f684af61aef783ee24
I20250109 20:53:11.526079 1166 ts_tablet_manager.cc:1939] T f1534c7240fc49f684af61aef783ee24 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.526878 1166 tablet_replica.cc:331] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.527576 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet f1534c7240fc49f684af61aef783ee24 (table default.Table_6c195088c8764cc2a94609872588c4a4 [id=97a35961c6c24ff6bee932ed763714b3]) successfully deleted
I20250109 20:53:11.502350 1158 tablet_bootstrap.cc:654] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.524698 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 3293f49190684b72a5c41031e621bd36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:03 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.524835 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 527776a5d742446eb8fe61c3c855021f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.524911 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c83e709a8d4e078e67af58ec51beac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.524963 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 88479a3f57654d84ad622af0782cce47 with delete_type TABLET_DATA_DELETED (replaced by tablet 43df4de52ddd4765984aee1c4d2479da) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.525065 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af9563f6efd47059caac7c86c41d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:00 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.532902 1166 raft_consensus.cc:2238] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.533303 1166 raft_consensus.cc:2267] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.533885 1166 ts_tablet_manager.cc:1905] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.536132 1157 raft_consensus.cc:2267] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.537940 1157 ts_tablet_manager.cc:1905] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.539839 1160 ts_tablet_manager.cc:1918] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.543834 1166 ts_tablet_manager.cc:1918] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.541200 1158 tablet_bootstrap.cc:492] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.545475 1158 ts_tablet_manager.cc:1397] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.000s
I20250109 20:53:11.547070 1158 raft_consensus.cc:357] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.547391 1158 raft_consensus.cc:383] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.547456 1158 raft_consensus.cc:738] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.547631 1158 consensus_queue.cc:260] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.552688 1158 ts_tablet_manager.cc:1428] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250109 20:53:11.556545 1158 tablet_bootstrap.cc:492] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.540226 1165 raft_consensus.cc:357] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.557281 1165 raft_consensus.cc:383] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.557355 1165 raft_consensus.cc:738] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.557287 1160 log.cc:1198] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/b74a4c1bdd8f4f6e9592ee5d97c58cc6
I20250109 20:53:11.557572 1165 consensus_queue.cc:260] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.558005 1165 ts_tablet_manager.cc:1428] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.082s user 0.001s sys 0.001s
I20250109 20:53:11.558825 1158 tablet_bootstrap.cc:654] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.557682 1160 ts_tablet_manager.cc:1939] T b74a4c1bdd8f4f6e9592ee5d97c58cc6 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:11.562717 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_29a47488e34648d2ab3f4c510af1ad3e" } new_table_name: "default.Table_426b90793d3a47ada79acdd8c9031599" modify_external_catalogs: true
I20250109 20:53:11.564309 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2883eb1cae054dc4b3073c4944e5355f" } modify_external_catalogs: true
I20250109 20:53:11.564460 1166 log.cc:1198] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/c83b61e879fc4979a85c872f2e6ab88c
I20250109 20:53:11.564462 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_2883eb1cae054dc4b3073c4944e5355f" } modify_external_catalogs: true
I20250109 20:53:11.564797 1166 ts_tablet_manager.cc:1939] T c83b61e879fc4979a85c872f2e6ab88c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:53:11.565441 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce5db9c45b24ce7a7731293b3637bb9 consensus state has no leader
I20250109 20:53:11.565829 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c2309577bfff4bfe9b2e5c15757ba2cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:11.566382 1166 tablet_replica.cc:331] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.566550 1166 raft_consensus.cc:2238] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.566670 1166 raft_consensus.cc:2267] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.567222 1165 tablet_bootstrap.cc:492] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.567221 1166 ts_tablet_manager.cc:1905] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.567237 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7e276816fdaf4a70903bc13b0d7863fe" } new_table_name: "default.Table_4b88260ae10c4d17ac67c34d892225cd" modify_external_catalogs: true
I20250109 20:53:11.568534 1165 tablet_bootstrap.cc:654] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.572254 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet bfa3e55fc83243a3b4978e981fdadf8c
I20250109 20:53:11.572909 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c4bf8dafa0e24de5a186f5cee909662f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:11.573393 885 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa3e55fc83243a3b4978e981fdadf8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.570312 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet c83b61e879fc4979a85c872f2e6ab88c (table default.Table_5cf48a44b13a4059b1e8974875e736e2 [id=7080faa7b3844702859c8a48be032b6d]) successfully deleted
W20250109 20:53:11.575775 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 76e688003f5b4a28adfcba0a4ce4b9da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e688003f5b4a28adfcba0a4ce4b9da already in progress: creating tablet
I20250109 20:53:11.576896 752 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa3e55fc83243a3b4978e981fdadf8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:11.577389 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4bf8dafa0e24de5a186f5cee909662f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.578799 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa3e55fc83243a3b4978e981fdadf8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:11.579144 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2309577bfff4bfe9b2e5c15757ba2cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.582952 1166 ts_tablet_manager.cc:1918] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.583969 885 tablet_service.cc:1514] Processing DeleteTablet for tablet da7f67026b164e0faad128d34188afb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.585060 1160 tablet_replica.cc:331] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.585186 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet b74a4c1bdd8f4f6e9592ee5d97c58cc6 (table default.Table_7ab416cb67944fa5b70140a8b8ab5e45 [id=5f7ff88269fa48ec80fe384008ba5780]) successfully deleted
I20250109 20:53:11.585273 1160 raft_consensus.cc:2238] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.585399 1160 raft_consensus.cc:2267] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.585937 1160 ts_tablet_manager.cc:1905] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.586601 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cc75a32bf9e14c8ba94083faff39f161"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.587034 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc75a32bf9e14c8ba94083faff39f161 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.592355 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_00dd367ba48b4c8c93d19c4ec1d7017c" } modify_external_catalogs: true
I20250109 20:53:11.592515 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_00dd367ba48b4c8c93d19c4ec1d7017c" } modify_external_catalogs: true
I20250109 20:53:11.596310 1165 tablet_bootstrap.cc:492] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.596436 1165 ts_tablet_manager.cc:1397] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.007s
I20250109 20:53:11.597414 1165 raft_consensus.cc:357] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.597592 1165 raft_consensus.cc:383] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.597633 1165 raft_consensus.cc:738] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.597803 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d13a0a21f1cd4b9db09a5f9d727cc382" } modify_external_catalogs: true
I20250109 20:53:11.597795 1165 consensus_queue.cc:260] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.597934 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d13a0a21f1cd4b9db09a5f9d727cc382" } modify_external_catalogs: true
I20250109 20:53:11.598224 1165 ts_tablet_manager.cc:1428] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:11.598436 1165 tablet_bootstrap.cc:492] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.596750 1176 tablet_bootstrap.cc:492] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.599951 1176 ts_tablet_manager.cc:1397] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.106s user 0.010s sys 0.023s
I20250109 20:53:11.600757 1176 raft_consensus.cc:357] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.601004 1176 raft_consensus.cc:383] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.601081 1176 raft_consensus.cc:738] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.601315 1176 consensus_queue.cc:260] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.602072 1176 ts_tablet_manager.cc:1428] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:11.602263 1166 log.cc:1198] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/edc803f2936b4538a5e70ef34e5cc5d4
I20250109 20:53:11.603405 1166 ts_tablet_manager.cc:1939] T edc803f2936b4538a5e70ef34e5cc5d4 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.605899 1157 ts_tablet_manager.cc:1918] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.606108 1157 log.cc:1198] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/6461f153a5994412ba05c3b687963fd1
I20250109 20:53:11.606532 1157 ts_tablet_manager.cc:1939] T 6461f153a5994412ba05c3b687963fd1 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:11.611564 2835 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:11.612021 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 6461f153a5994412ba05c3b687963fd1 (table default.Table_81566c8f4bd940aab3c5fca6ab8ff292 [id=37978bfd578840e4962df1552f7db45a]) successfully deleted
I20250109 20:53:11.613284 1157 tablet_replica.cc:331] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:11.614130 1157 raft_consensus.cc:2238] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.617314 1157 raft_consensus.cc:2267] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.621208 2826 raft_consensus.cc:491] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.621346 2826 raft_consensus.cc:513] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.621953 2826 leader_election.cc:290] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.622267 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0860554cac64b8882e39fa62462b1aa" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:11.622440 777 raft_consensus.cc:2463] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:53:11.622444 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0860554cac64b8882e39fa62462b1aa" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:11.622793 990 leader_election.cc:304] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
W20250109 20:53:11.622978 989 leader_election.cc:343] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.623314 2826 raft_consensus.cc:2798] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:11.623373 2826 raft_consensus.cc:491] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:11.623427 2826 raft_consensus.cc:3054] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:11.624233 1157 ts_tablet_manager.cc:1905] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.629606 2838 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:11.631071 1050 raft_consensus.cc:1270] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Refusing update from remote peer 09833ea46b7141d58c3899dace69e49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250109 20:53:11.631696 712 consensus_peers.cc:487] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:11.631976 1176 tablet_bootstrap.cc:492] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.632177 1166 tablet_replica.cc:331] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.632390 1166 raft_consensus.cc:2238] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.632513 1166 raft_consensus.cc:2267] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.633252 2692 consensus_queue.cc:1035] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:11.633466 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet edc803f2936b4538a5e70ef34e5cc5d4 (table default.Table_f456c719d70549da87d54647fd2d02aa [id=d969b9f61388495f90e0351621b658e2]) successfully deleted
I20250109 20:53:11.637876 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 048be84b36a1450a818f05fb36cbf3b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:06 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.638175 1165 tablet_bootstrap.cc:654] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.638221 1166 ts_tablet_manager.cc:1905] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.641919 1160 ts_tablet_manager.cc:1918] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.643347 1176 tablet_bootstrap.cc:654] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.625967 2826 raft_consensus.cc:513] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.646245 2826 leader_election.cc:290] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.646579 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0860554cac64b8882e39fa62462b1aa" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:11.646787 777 raft_consensus.cc:3054] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:11.647625 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7318e12d39b649ceb1971008c72c893b" } modify_external_catalogs: true
I20250109 20:53:11.647778 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7318e12d39b649ceb1971008c72c893b" } modify_external_catalogs: true
I20250109 20:53:11.648833 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7c925d21560f4a888144a3c8bc75a2c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.649382 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c925d21560f4a888144a3c8bc75a2c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.649381 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0860554cac64b8882e39fa62462b1aa" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:11.649717 1160 log.cc:1198] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/6c4abc19297e4029b1b7c5396b49b967
I20250109 20:53:11.649901 1166 ts_tablet_manager.cc:1918] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.650002 1160 ts_tablet_manager.cc:1939] T 6c4abc19297e4029b1b7c5396b49b967 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:11.650002 1166 log.cc:1198] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/40546e1b5e854a79a7f706a43b6c6062
I20250109 20:53:11.650230 1166 ts_tablet_manager.cc:1939] T 40546e1b5e854a79a7f706a43b6c6062 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.650383 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e556f33cc6824d0cb0f84e33ae69638e" } new_table_name: "default.Table_89e2bb42f14d4cbbbeacdfe857cd1fe2" modify_external_catalogs: true
I20250109 20:53:11.650501 1160 tablet_replica.cc:331] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.650652 1160 raft_consensus.cc:2238] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.650766 1160 raft_consensus.cc:2267] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.650960 1166 tablet_replica.cc:331] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.651105 1166 raft_consensus.cc:2238] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.651202 1160 ts_tablet_manager.cc:1905] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.651213 1166 raft_consensus.cc:2267] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.652295 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 40546e1b5e854a79a7f706a43b6c6062 (table default.Table_6c79e0eceb7b4c6a8c751387a1a7118a [id=3b1b495828aa4de2b1362a99304baedc]) successfully deleted
I20250109 20:53:11.654089 777 raft_consensus.cc:2463] T c0860554cac64b8882e39fa62462b1aa P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:11.654554 990 leader_election.cc:304] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:11.655507 2822 raft_consensus.cc:2798] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:11.655608 2822 raft_consensus.cc:695] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:53:11.655827 2822 consensus_queue.cc:237] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.657182 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e5b00a1d5f624d3fb6ff7d93ccf66815"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.657622 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5b00a1d5f624d3fb6ff7d93ccf66815 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.657672 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 6c4abc19297e4029b1b7c5396b49b967 (table default.Table_eff4d27a17b3425c92c3d388ee1cc73e [id=6f77cddda4de4f6083cedc70e3026b6d]) successfully deleted
W20250109 20:53:11.657877 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 76e688003f5b4a28adfcba0a4ce4b9da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e688003f5b4a28adfcba0a4ce4b9da already in progress: creating tablet
W20250109 20:53:11.658125 989 leader_election.cc:343] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.661927 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3c5e9b31b27949c2948cb64b6124ee72
I20250109 20:53:11.663178 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5e9b31b27949c2948cb64b6124ee72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.663344 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5e9b31b27949c2948cb64b6124ee72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.663998 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5e9b31b27949c2948cb64b6124ee72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.667289 1166 ts_tablet_manager.cc:1905] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:11.670738 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b71604f6b7f6423a8aa8618989a1f5b8 consensus state has no leader
I20250109 20:53:11.671782 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_5c4a7abac8a742b697104214d095c2d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.672338 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c4a7abac8a742b697104214d095c2d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.674525 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7afd8872782446a39a98ace49db8ebb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.674985 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7afd8872782446a39a98ace49db8ebb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.685834 1160 ts_tablet_manager.cc:1918] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.692050 1160 log.cc:1198] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/006475691733470f8a0468365972dfd2
I20250109 20:53:11.697031 1166 ts_tablet_manager.cc:1918] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.697144 1166 log.cc:1198] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/cbef1d27c5394fc1b6354c5d6f3a8f1f
I20250109 20:53:11.697400 2822 raft_consensus.cc:491] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.697526 2822 raft_consensus.cc:513] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.698132 2822 leader_election.cc:290] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.698628 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64e619e3e7e49469adca01d95d75d02" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:11.699084 990 leader_election.cc:343] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.697412 1166 ts_tablet_manager.cc:1939] T cbef1d27c5394fc1b6354c5d6f3a8f1f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.701196 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet cbef1d27c5394fc1b6354c5d6f3a8f1f (table default.Table_54acd558d6f04df09ecc0669cabfe878 [id=81f954b2a3874b34b98f34d39a126140]) successfully deleted
W20250109 20:53:11.702234 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5f55dd45ad16489cbfaa59338a2fbb44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f55dd45ad16489cbfaa59338a2fbb44 already in progress: creating tablet
W20250109 20:53:11.702965 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 5bfa04c33e104c148a73c545284c20a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bfa04c33e104c148a73c545284c20a1 already in progress: creating tablet
W20250109 20:53:11.703603 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 05df026727da4ea594810bb9b516dc7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05df026727da4ea594810bb9b516dc7c already in progress: creating tablet
W20250109 20:53:11.704234 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 2b3af68b009a4789bc1cdbafc8d44dcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b3af68b009a4789bc1cdbafc8d44dcf already in progress: creating tablet
I20250109 20:53:11.704571 2822 raft_consensus.cc:491] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.704670 2822 raft_consensus.cc:513] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.705207 2822 leader_election.cc:290] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.705672 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418e1bd414914b21a233c3099e2057da" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:11.706274 990 leader_election.cc:343] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:11.709321 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 8e67aa0fafa44a75bceeaff2f05298bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e67aa0fafa44a75bceeaff2f05298bc already in progress: creating tablet
I20250109 20:53:11.709818 1166 tablet_replica.cc:331] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.710177 1166 raft_consensus.cc:2238] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.710520 1166 raft_consensus.cc:2267] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.696341 1158 tablet_bootstrap.cc:492] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.716084 1158 ts_tablet_manager.cc:1397] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.160s user 0.006s sys 0.006s
I20250109 20:53:11.716820 1158 raft_consensus.cc:357] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.717010 1158 raft_consensus.cc:383] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.717064 1158 raft_consensus.cc:738] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.721940 1158 consensus_queue.cc:260] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.693801 2788 tablet.cc:1721] T 2608f48f5950466e9ab2d12ee87503cb P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250109 20:53:11.708700 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418e1bd414914b21a233c3099e2057da" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:11.723341 1160 ts_tablet_manager.cc:1939] T 006475691733470f8a0468365972dfd2 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:53:11.723657 989 leader_election.cc:343] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.723765 989 leader_election.cc:304] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.724102 2822 raft_consensus.cc:2743] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.724346 1160 tablet_replica.cc:331] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.724524 1160 raft_consensus.cc:2238] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.724639 1160 raft_consensus.cc:2267] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.724859 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 006475691733470f8a0468365972dfd2 (table default.Table_d5db66989ced4d6fb84352f19e087cc7 [id=d65cb2a049bb450391b2f32bb6bbeead]) successfully deleted
I20250109 20:53:11.725253 1158 ts_tablet_manager.cc:1428] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
W20250109 20:53:11.725287 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5f55dd45ad16489cbfaa59338a2fbb44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f55dd45ad16489cbfaa59338a2fbb44 already in progress: creating tablet
I20250109 20:53:11.725489 1158 tablet_bootstrap.cc:492] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:11.725467 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 5bfa04c33e104c148a73c545284c20a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bfa04c33e104c148a73c545284c20a1 already in progress: creating tablet
W20250109 20:53:11.725633 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 05df026727da4ea594810bb9b516dc7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05df026727da4ea594810bb9b516dc7c already in progress: creating tablet
W20250109 20:53:11.725806 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 2b3af68b009a4789bc1cdbafc8d44dcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b3af68b009a4789bc1cdbafc8d44dcf already in progress: creating tablet
I20250109 20:53:11.725944 2822 raft_consensus.cc:491] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:53:11.726003 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 8e67aa0fafa44a75bceeaff2f05298bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e67aa0fafa44a75bceeaff2f05298bc already in progress: creating tablet
I20250109 20:53:11.726068 2822 raft_consensus.cc:513] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:11.726186 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 88479a3f57654d84ad622af0782cce47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88479a3f57654d84ad622af0782cce47 already in progress: creating tablet
I20250109 20:53:11.724756 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fe9641b0cb0b43dc9a3d40646c1f7526"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:11.726646 2822 leader_election.cc:290] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
W20250109 20:53:11.726877 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe9641b0cb0b43dc9a3d40646c1f7526 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.728915 2835 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:11.730065 1166 ts_tablet_manager.cc:1905] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.730523 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50039da210df47ab8476e28387c8a802" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:11.730794 2788 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250109 20:53:11.732692 990 leader_election.cc:343] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.733364 1160 ts_tablet_manager.cc:1905] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.733819 1158 tablet_bootstrap.cc:654] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.737159 2838 tablet.cc:1721] T cda300a7e58a44339c3bdb2e62a9f384 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:11.741573 1158 tablet_bootstrap.cc:492] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.741683 1158 ts_tablet_manager.cc:1397] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250109 20:53:11.742632 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50039da210df47ab8476e28387c8a802" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:11.743750 1158 raft_consensus.cc:357] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.743916 1158 raft_consensus.cc:383] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.743969 1158 raft_consensus.cc:738] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.744154 1158 consensus_queue.cc:260] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:11.744364 989 leader_election.cc:343] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.744469 989 leader_election.cc:304] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.744570 1158 ts_tablet_manager.cc:1428] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250109 20:53:11.744832 1158 tablet_bootstrap.cc:492] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.744868 2822 raft_consensus.cc:2743] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.746091 2835 tablet.cc:1721] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:11.746361 1158 tablet_bootstrap.cc:654] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.747570 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_87f3dbf3f44842bfa78405e88b9b411b [id=17d12fac83d6463fb91fab6729a08139] alter complete (version 4)
I20250109 20:53:11.750320 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64e619e3e7e49469adca01d95d75d02" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:11.750684 2788 tablet.cc:1721] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250109 20:53:11.750783 989 leader_election.cc:343] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.750902 989 leader_election.cc:304] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.751319 2822 raft_consensus.cc:2743] T a64e619e3e7e49469adca01d95d75d02 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.756419 914 raft_consensus.cc:3054] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:11.760906 1166 ts_tablet_manager.cc:1918] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.763059 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_77cd8321243641f3970bbeda1be552e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.763871 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77cd8321243641f3970bbeda1be552e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.765430 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fbcb20c87fa34c8cb9c40e7098d01395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.765969 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbcb20c87fa34c8cb9c40e7098d01395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.766177 1166 log.cc:1198] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/22f6ec8f3d4f4d37a11f15173449155c
I20250109 20:53:11.766355 2807 raft_consensus.cc:491] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.766523 2807 raft_consensus.cc:513] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.766710 1166 ts_tablet_manager.cc:1939] T 22f6ec8f3d4f4d37a11f15173449155c P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.766566 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_107c6d6e3acc430f94405d1287423d77" } new_table_name: "default.Table_44f61aadccac4df580d251c592fc2436" modify_external_catalogs: true
I20250109 20:53:11.767168 2807 leader_election.cc:290] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.768325 1160 ts_tablet_manager.cc:1918] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.766846 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b652be7142be44bab45b5ad83a328a06" } new_table_name: "default.Table_7fe5334074314521aa862443562bf437" modify_external_catalogs: true
I20250109 20:53:11.770707 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f4bdda16554d9d81de5ca5c5205f4c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:11.770793 2835 tablet.cc:1721] T cc54e97e91c54afea098624bd85fcc6e P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:11.770941 1051 raft_consensus.cc:2463] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:11.771332 713 leader_election.cc:304] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:11.771561 2692 raft_consensus.cc:2798] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:11.771639 2692 raft_consensus.cc:491] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:11.771697 2692 raft_consensus.cc:3054] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:11.773731 2692 raft_consensus.cc:513] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.776343 2692 leader_election.cc:290] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.777374 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_d0d17c135e164e9688c77b42de045b4e" } new_table_name: "default.Table_59cdca01da47471ab4604a4d1863606e" modify_external_catalogs: true
I20250109 20:53:11.777483 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f4bdda16554d9d81de5ca5c5205f4c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:11.777637 1051 raft_consensus.cc:3054] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:11.780026 1051 raft_consensus.cc:2463] T 74f4bdda16554d9d81de5ca5c5205f4c P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:11.780710 713 leader_election.cc:304] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:11.781315 2695 raft_consensus.cc:2798] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:11.781447 2695 raft_consensus.cc:695] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:11.781675 2695 consensus_queue.cc:237] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.785006 2838 tablet.cc:1721] T cc54e97e91c54afea098624bd85fcc6e P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250109 20:53:11.789553 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 22f6ec8f3d4f4d37a11f15173449155c (table default.Table_2bdba403f4a24ce392324eeb74d3ed0e [id=ab5d37b49fc549f8a7829a858ea0b3bf]) successfully deleted
I20250109 20:53:11.789810 1160 log.cc:1198] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/e716270ffcf44039b075023fd0c2d2a3
I20250109 20:53:11.790160 1160 ts_tablet_manager.cc:1939] T e716270ffcf44039b075023fd0c2d2a3 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:11.790719 1160 tablet_replica.cc:331] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.790899 1160 raft_consensus.cc:2238] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.791033 1160 raft_consensus.cc:2267] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.791288 1166 tablet_replica.cc:331] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
W20250109 20:53:11.791456 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6df8eda5c5f4c25a1f7ad7e18f18d2d consensus state has no leader
I20250109 20:53:11.791457 1166 raft_consensus.cc:2238] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.791582 1166 raft_consensus.cc:2267] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.791819 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_463c5bd1d860495f81699d4a3a47a4c5" } new_table_name: "default.Table_cd561e7e29fc4285bcae3ddcc64c2732" modify_external_catalogs: true
I20250109 20:53:11.792287 1166 ts_tablet_manager.cc:1905] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.793053 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet e716270ffcf44039b075023fd0c2d2a3 (table default.Table_45d77a3e61d14f468ad3e271195a1cd9 [id=c96aada642b84c20894926b34fbea386]) successfully deleted
W20250109 20:53:11.793285 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet bfa3e55fc83243a3b4978e981fdadf8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfa3e55fc83243a3b4978e981fdadf8c already in progress: creating tablet
I20250109 20:53:11.794906 1160 ts_tablet_manager.cc:1905] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:11.796094 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e82cda1841c24bb2a14c127b1f40ad26 consensus state has no leader
I20250109 20:53:11.796380 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_80bd0d8dd57e4f28a7edde0d3e630867" } modify_external_catalogs: true
I20250109 20:53:11.796514 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_80bd0d8dd57e4f28a7edde0d3e630867" } modify_external_catalogs: true
W20250109 20:53:11.799271 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab5bfc387bc24feeb73bf5ce44af27d0 consensus state has no leader
I20250109 20:53:11.800489 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_ad38a22cf3f54e9b85367979db2c2d21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.801101 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad38a22cf3f54e9b85367979db2c2d21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.801148 1158 tablet_bootstrap.cc:492] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.801287 1158 ts_tablet_manager.cc:1397] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.001s
W20250109 20:53:11.802027 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c785c74ecee84e6fb5d27560cc7c6f28 consensus state has no leader
I20250109 20:53:11.802007 1158 raft_consensus.cc:357] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.802212 1158 raft_consensus.cc:383] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.802285 1158 raft_consensus.cc:738] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.802361 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_54fde017f7de4545a00254c29d41224d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:11.802486 1158 consensus_queue.cc:260] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:11.802917 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54fde017f7de4545a00254c29d41224d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.803025 1158 ts_tablet_manager.cc:1428] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:11.803272 1158 tablet_bootstrap.cc:492] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.780431 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f4bdda16554d9d81de5ca5c5205f4c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:11.788993 1165 tablet_bootstrap.cc:492] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.805315 1165 ts_tablet_manager.cc:1397] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.207s user 0.011s sys 0.014s
W20250109 20:53:11.805630 712 leader_election.cc:343] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.806030 1165 raft_consensus.cc:357] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.806223 1165 raft_consensus.cc:383] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.806282 1165 raft_consensus.cc:738] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.806514 1165 consensus_queue.cc:260] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.807004 1165 ts_tablet_manager.cc:1428] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250109 20:53:11.807278 1165 tablet_bootstrap.cc:492] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.808933 1165 tablet_bootstrap.cc:654] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.809672 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0a598d1333b34029ae6c012b16c49f83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.810281 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a598d1333b34029ae6c012b16c49f83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.810693 1158 tablet_bootstrap.cc:654] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.814043 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f4bdda16554d9d81de5ca5c5205f4c" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.815121 712 leader_election.cc:343] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.815939 1166 ts_tablet_manager.cc:1918] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.819168 2695 raft_consensus.cc:491] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.819353 2695 raft_consensus.cc:513] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.820367 2695 leader_election.cc:290] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:11.821130 713 leader_election.cc:343] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 527776a5d742446eb8fe61c3c855021f
I20250109 20:53:11.822409 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527776a5d742446eb8fe61c3c855021f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:11.822945 712 leader_election.cc:343] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.823067 712 leader_election.cc:304] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.823104 1166 log.cc:1198] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/80faa29a3b314796b68a4073665d2898
I20250109 20:53:11.823410 1166 ts_tablet_manager.cc:1939] T 80faa29a3b314796b68a4073665d2898 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.823530 2695 raft_consensus.cc:2743] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.824888 1160 ts_tablet_manager.cc:1918] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.826354 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 80faa29a3b314796b68a4073665d2898 (table default.Table_283db7be437c43d8845b9f6d4d906bc7 [id=981a80b153d644bcab94aaaaa9aa6e39]) successfully deleted
W20250109 20:53:11.828361 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 88479a3f57654d84ad622af0782cce47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88479a3f57654d84ad622af0782cce47 already in progress: creating tablet
I20250109 20:53:11.828857 1160 log.cc:1198] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/da7f67026b164e0faad128d34188afb5
W20250109 20:53:11.829437 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet bfa3e55fc83243a3b4978e981fdadf8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfa3e55fc83243a3b4978e981fdadf8c already in progress: creating tablet
I20250109 20:53:11.829838 1157 ts_tablet_manager.cc:1918] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.830015 1160 ts_tablet_manager.cc:1939] T da7f67026b164e0faad128d34188afb5 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:11.830296 1157 log.cc:1198] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/face19003d0042669e3f22553f9fe19a
I20250109 20:53:11.830396 1166 tablet_replica.cc:331] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.830663 1157 ts_tablet_manager.cc:1939] T face19003d0042669e3f22553f9fe19a P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:11.831053 1166 raft_consensus.cc:2238] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.831612 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet face19003d0042669e3f22553f9fe19a (table default.Table_94a783531318417989d76ed0dd1d8bdc [id=2b3bd52f32a145fdb6228cf715245ed9]) successfully deleted
I20250109 20:53:11.831645 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet da7f67026b164e0faad128d34188afb5 (table default.Table_0191c74c02514d34a53c6db0d807f298 [id=b494cc0e89104a849325f838046bc82d]) successfully deleted
I20250109 20:53:11.832237 1166 raft_consensus.cc:2267] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.832640 1160 tablet_replica.cc:331] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.833372 1166 ts_tablet_manager.cc:1905] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.834965 1160 raft_consensus.cc:2238] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.835507 1160 raft_consensus.cc:2267] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.836494 1160 ts_tablet_manager.cc:1905] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.841931 2838 tablet.cc:1721] T 0b083e6be42f41ecbf23ee57ec17b0a5 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:11.851656 1166 ts_tablet_manager.cc:1918] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:11.854176 1160 ts_tablet_manager.cc:1918] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.855027 1166 log.cc:1198] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/3c5e9b31b27949c2948cb64b6124ee72
I20250109 20:53:11.855397 1166 ts_tablet_manager.cc:1939] T 3c5e9b31b27949c2948cb64b6124ee72 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:11.854940 1165 tablet_bootstrap.cc:492] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.855603 1165 ts_tablet_manager.cc:1397] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.011s
I20250109 20:53:11.856324 1165 raft_consensus.cc:357] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.856559 1165 raft_consensus.cc:383] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.856652 1165 raft_consensus.cc:738] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.857178 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 048be84b36a1450a818f05fb36cbf3b9 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:11.857131 1165 consensus_queue.cc:260] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.857299 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_426b90793d3a47ada79acdd8c9031599 [id=1284f08f96184576bde56fa61834e33f] alter complete (version 3)
I20250109 20:53:11.859421 1165 ts_tablet_manager.cc:1428] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:11.859648 1165 tablet_bootstrap.cc:492] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.862612 1165 tablet_bootstrap.cc:654] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.864120 1160 log.cc:1198] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/048be84b36a1450a818f05fb36cbf3b9
I20250109 20:53:11.864486 1160 ts_tablet_manager.cc:1939] T 048be84b36a1450a818f05fb36cbf3b9 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:11.865224 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 048be84b36a1450a818f05fb36cbf3b9 (table default.Table_c01a7fe9ae3947a59c88c37c98a59cab [id=adc7cde5fa0445c0a26fafa6c692f8c0]) successfully deleted
I20250109 20:53:11.865356 1158 tablet_bootstrap.cc:492] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.865497 1158 ts_tablet_manager.cc:1397] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.006s
I20250109 20:53:11.865950 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]) successfully deleted
I20250109 20:53:11.866318 1160 tablet_replica.cc:331] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:11.866286 1158 raft_consensus.cc:357] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.866492 1158 raft_consensus.cc:383] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.866427 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_662f7ce5426e4f55b64206655147514a" } modify_external_catalogs: true
I20250109 20:53:11.866573 1158 raft_consensus.cc:738] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.866596 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_662f7ce5426e4f55b64206655147514a" } modify_external_catalogs: true
I20250109 20:53:11.866766 1160 raft_consensus.cc:2238] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.866784 1158 consensus_queue.cc:260] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.866940 1160 raft_consensus.cc:2267] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.867129 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_cb9352dcbd274778a9b8f2055d1de2cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:11.867282 1158 ts_tablet_manager.cc:1428] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:11.867417 1160 ts_tablet_manager.cc:1905] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.867535 1158 tablet_bootstrap.cc:492] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:11.867592 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb9352dcbd274778a9b8f2055d1de2cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.869037 1158 tablet_bootstrap.cc:654] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.869711 2818 raft_consensus.cc:491] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.869814 2818 raft_consensus.cc:513] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.870322 2818 leader_election.cc:290] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:11.871271 854 leader_election.cc:343] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 9d7587432d7345e4a225fc7d5c968c27
W20250109 20:53:11.871722 854 leader_election.cc:343] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 9d7587432d7345e4a225fc7d5c968c27
I20250109 20:53:11.871799 854 leader_election.cc:304] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.872115 2818 raft_consensus.cc:2743] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.872947 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_4b88260ae10c4d17ac67c34d892225cd [id=68f5eeffe12b4cda81ebb874ada4dfe5] alter complete (version 1)
I20250109 20:53:11.873607 1157 tablet_replica.cc:331] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:11.875558 1157 raft_consensus.cc:2238] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.875710 1157 raft_consensus.cc:2267] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.876185 1157 ts_tablet_manager.cc:1905] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.876289 1165 tablet_bootstrap.cc:492] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.876386 1165 ts_tablet_manager.cc:1397] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250109 20:53:11.877020 1165 raft_consensus.cc:357] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.877199 1165 raft_consensus.cc:383] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.877257 1165 raft_consensus.cc:738] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.877460 1165 consensus_queue.cc:260] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.878139 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cfef853be74f4125ae5717179a124807" } new_table_name: "default.Table_4ab544cd23024498b1983ef3bd7042d5" modify_external_catalogs: true
I20250109 20:53:11.879617 1165 ts_tablet_manager.cc:1428] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:53:11.879855 1165 tablet_bootstrap.cc:492] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.881269 1176 tablet_bootstrap.cc:492] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.881412 1176 ts_tablet_manager.cc:1397] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.250s user 0.030s sys 0.033s
I20250109 20:53:11.882117 1176 raft_consensus.cc:357] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.882326 1176 raft_consensus.cc:383] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.882373 1176 raft_consensus.cc:738] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.882584 1176 consensus_queue.cc:260] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.885289 1158 tablet_bootstrap.cc:492] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:11.885435 1158 ts_tablet_manager.cc:1397] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250109 20:53:11.885672 450 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet a85c36689d8f4b16b85db5ad3c84cac0 (table default.Table_7e2171ec1bd442eb911f34154fbc9f58 [id=2586e7a951ca4dfc84297301bedbd5ba]). Expected version 1 got 0
I20250109 20:53:11.885914 1176 ts_tablet_manager.cc:1428] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250109 20:53:11.886044 2822 raft_consensus.cc:491] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.886200 1176 tablet_bootstrap.cc:492] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.886147 2822 raft_consensus.cc:513] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.886487 1158 raft_consensus.cc:357] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.886674 2822 leader_election.cc:290] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.886699 1158 raft_consensus.cc:383] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.886772 1158 raft_consensus.cc:738] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.886992 1158 consensus_queue.cc:260] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.887265 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7029146f2854a89999e87961db6b2e3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:11.887517 1158 ts_tablet_manager.cc:1428] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:11.887820 1158 tablet_bootstrap.cc:492] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:11.887831 1176 tablet_bootstrap.cc:654] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.888171 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7029146f2854a89999e87961db6b2e3" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:11.888832 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 2769e5906c604bd5b14df79db765ec61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:10 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:11.889074 989 leader_election.cc:343] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:11.889496 990 leader_election.cc:343] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.889591 990 leader_election.cc:304] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:11.890304 2818 raft_consensus.cc:491] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.890406 2818 raft_consensus.cc:513] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.890969 2818 leader_election.cc:290] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:11.892040 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7587432d7345e4a225fc7d5c968c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:11.892278 854 leader_election.cc:343] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2769e5906c604bd5b14df79db765ec61
W20250109 20:53:11.892434 854 leader_election.cc:343] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 2769e5906c604bd5b14df79db765ec61
I20250109 20:53:11.892506 854 leader_election.cc:304] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:11.893064 2826 raft_consensus.cc:2743] T f7029146f2854a89999e87961db6b2e3 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.894169 1158 tablet_bootstrap.cc:654] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.894145 2818 raft_consensus.cc:2743] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:11.895393 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4870a667cd3e410590923c9e2497af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.908766 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_bc61fb7c754b48f2868416a0158388f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:11.908896 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_96ef863aeef94bf5950c9be167dde185"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.909299 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96ef863aeef94bf5950c9be167dde185 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:11.909299 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc61fb7c754b48f2868416a0158388f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.910434 1165 tablet_bootstrap.cc:654] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:11.917707 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6434cbc16eb47879089491e19d8bff6 has no consensus state
I20250109 20:53:11.918051 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_188ed9bb91204948b8552c0b799b00d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.918509 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_188ed9bb91204948b8552c0b799b00d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.920073 1160 ts_tablet_manager.cc:1918] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250109 20:53:11.922902 1165 tablet_bootstrap.cc:492] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:11.923664 1160 log.cc:1198] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/3c5e9b31b27949c2948cb64b6124ee72
I20250109 20:53:11.923674 1165 ts_tablet_manager.cc:1397] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.003s
I20250109 20:53:11.925148 1160 ts_tablet_manager.cc:1939] T 3c5e9b31b27949c2948cb64b6124ee72 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:11.925511 1165 raft_consensus.cc:357] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.925720 1165 raft_consensus.cc:383] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.925730 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 3c5e9b31b27949c2948cb64b6124ee72 (table default.Table_7318e12d39b649ceb1971008c72c893b [id=d320c4ac79d046479cf0718148b62b3b]) successfully deleted
I20250109 20:53:11.925796 1165 raft_consensus.cc:738] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.925988 1165 consensus_queue.cc:260] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.926469 1165 ts_tablet_manager.cc:1428] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:11.927331 1165 tablet_bootstrap.cc:492] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:11.928857 1165 tablet_bootstrap.cc:654] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.930539 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet ee208bad76734a8fa1ada438bdbbe48c (table default.Table_79fb13be9f0440c9991b67d641ab0b6a [id=238aa3a3a8e74dfba23e60c7d6f049cf]). Expected version 1 got 0
I20250109 20:53:11.930722 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 8a45cd5ffe7040e385d8cdfb97ed011a (table default.Table_4ef8244295a842d5b998b18f6cf71e64 [id=4d3659cafa3246f1a84d6a4ddd44f82f]). Expected version 4 got 0
I20250109 20:53:11.930822 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 74f4bdda16554d9d81de5ca5c5205f4c (table default.Table_d2d8f346b1d2494cb588d606237080a0 [id=dfdbf927d4ff4f329975148ef22d8940]). Expected version 1 got 0
I20250109 20:53:11.932029 451 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet beee9825fa0240e88029bf32ace0e187 (table default.Table_702ee9d09df94d92bbd7718ccdb1a590 [id=be2676e412c143c39fd150f5703f6f48]). Expected version 1 got 0
I20250109 20:53:11.932024 885 tablet_service.cc:1467] Processing CreateTablet for tablet 168dffba70074ce6b80ec2179d6f6b72 (DEFAULT_TABLE table=default.Table_bc12f5dd82b04974b89e88ba08f752b9 [id=2e6e5c55b2bf4cbabfa522cab7da9aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.932403 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 168dffba70074ce6b80ec2179d6f6b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.937960 742 tablet_service.cc:1467] Processing CreateTablet for tablet 168dffba70074ce6b80ec2179d6f6b72 (DEFAULT_TABLE table=default.Table_bc12f5dd82b04974b89e88ba08f752b9 [id=2e6e5c55b2bf4cbabfa522cab7da9aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.938361 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 168dffba70074ce6b80ec2179d6f6b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.938351 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 168dffba70074ce6b80ec2179d6f6b72 (DEFAULT_TABLE table=default.Table_bc12f5dd82b04974b89e88ba08f752b9 [id=2e6e5c55b2bf4cbabfa522cab7da9aa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.938517 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 8926204e314f486dbdb7d0b14538b547 (DEFAULT_TABLE table=default.Table_b0b9646173584ad1b385b18ea22ad049 [id=9d6e6487b3564d6fa67f07ef334e4408]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.938694 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 168dffba70074ce6b80ec2179d6f6b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.938944 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 25a05f0e1e09412885544456315704e1 (DEFAULT_TABLE table=default.Table_42e8514aea2f48edb963804d6577aa3d [id=59b816f17f3542c7adea95f136fceac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.939244 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a05f0e1e09412885544456315704e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.940946 740 tablet_service.cc:1467] Processing CreateTablet for tablet 8926204e314f486dbdb7d0b14538b547 (DEFAULT_TABLE table=default.Table_b0b9646173584ad1b385b18ea22ad049 [id=9d6e6487b3564d6fa67f07ef334e4408]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.941275 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8926204e314f486dbdb7d0b14538b547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.943034 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 418e1bd414914b21a233c3099e2057da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.943245 1166 tablet_replica.cc:331] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:11.943425 1166 raft_consensus.cc:2238] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.943554 1166 raft_consensus.cc:2267] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.943935 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8926204e314f486dbdb7d0b14538b547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.944988 752 tablet_service.cc:1467] Processing CreateTablet for tablet 25a05f0e1e09412885544456315704e1 (DEFAULT_TABLE table=default.Table_42e8514aea2f48edb963804d6577aa3d [id=59b816f17f3542c7adea95f136fceac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.945307 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a05f0e1e09412885544456315704e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.945696 1176 tablet_bootstrap.cc:492] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:11.945813 1176 ts_tablet_manager.cc:1397] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.001s
I20250109 20:53:11.946502 1176 raft_consensus.cc:357] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.946676 1176 raft_consensus.cc:383] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:11.946720 1176 raft_consensus.cc:738] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:11.946898 1176 consensus_queue.cc:260] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.947482 1176 ts_tablet_manager.cc:1428] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:11.947695 1176 tablet_bootstrap.cc:492] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:11.949257 895 tablet_service.cc:1514] Processing DeleteTablet for tablet fce062dc0ac144a19f8fa3eef4a099b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:11.949342 898 tablet_service.cc:1467] Processing CreateTablet for tablet 8926204e314f486dbdb7d0b14538b547 (DEFAULT_TABLE table=default.Table_b0b9646173584ad1b385b18ea22ad049 [id=9d6e6487b3564d6fa67f07ef334e4408]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.949646 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8926204e314f486dbdb7d0b14538b547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.950234 1157 ts_tablet_manager.cc:1918] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:11.950460 1157 log.cc:1198] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/2797e1df3d12476f89fb5155523c3935
I20250109 20:53:11.950891 1157 ts_tablet_manager.cc:1939] T 2797e1df3d12476f89fb5155523c3935 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:11.952360 1157 tablet_replica.cc:331] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:11.952607 1157 raft_consensus.cc:2238] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:11.952986 892 tablet_service.cc:1467] Processing CreateTablet for tablet 25a05f0e1e09412885544456315704e1 (DEFAULT_TABLE table=default.Table_42e8514aea2f48edb963804d6577aa3d [id=59b816f17f3542c7adea95f136fceac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.953289 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a05f0e1e09412885544456315704e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.953392 1157 raft_consensus.cc:2267] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:11.953938 1157 ts_tablet_manager.cc:1905] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.957698 1166 ts_tablet_manager.cc:1905] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:11.961583 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 2797e1df3d12476f89fb5155523c3935 (table default.Table_ad0a7364cd0d48caa4d583519597f8fe [id=2b35e26fc97542b89dd2fa89fecf1206]) successfully deleted
W20250109 20:53:11.961879 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 5bfa04c33e104c148a73c545284c20a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bfa04c33e104c148a73c545284c20a1 already in progress: creating tablet
W20250109 20:53:11.962054 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 05df026727da4ea594810bb9b516dc7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05df026727da4ea594810bb9b516dc7c already in progress: creating tablet
W20250109 20:53:11.962213 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 2b3af68b009a4789bc1cdbafc8d44dcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b3af68b009a4789bc1cdbafc8d44dcf already in progress: creating tablet
W20250109 20:53:11.962359 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 8e67aa0fafa44a75bceeaff2f05298bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e67aa0fafa44a75bceeaff2f05298bc already in progress: creating tablet
W20250109 20:53:11.968251 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.60:41583 (error 108)
I20250109 20:53:11.968461 2822 raft_consensus.cc:491] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:11.968559 2822 raft_consensus.cc:513] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:11.969158 2822 leader_election.cc:290] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:11.969921 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5609eaf86d4e8ea3d3da41243c927f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:11.970465 990 leader_election.cc:343] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:11.977774 752 tablet_service.cc:1467] Processing CreateTablet for tablet 99facf2f6a7c4afc96ff0e06b58da937 (DEFAULT_TABLE table=default.Table_025053d76dd84c2a9a5d320fa3a085d9 [id=ee89624f7ec5426f92789da48e8dde27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.977965 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 81440079bca14ec08ed6716b786a4f73 (DEFAULT_TABLE table=default.Table_8252859a8fcd44b0984a8c054e3c0416 [id=47f6f9f7e158477aa8a86da4340ee906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.978293 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81440079bca14ec08ed6716b786a4f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.979001 1025 tablet_service.cc:1467] Processing CreateTablet for tablet e32567235f7e49b6958a822e4836409b (DEFAULT_TABLE table=default.Table_1d3219f0687d4cb595ce0dae72cfb41a [id=bc65fa5b7dd94ccfaa3d89164a946988]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.979247 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32567235f7e49b6958a822e4836409b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.980324 1176 tablet_bootstrap.cc:654] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:11.980487 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9581bb46aecf427eb75a633db2adda3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:11.980991 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9581bb46aecf427eb75a633db2adda3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.978438 739 tablet_service.cc:1467] Processing CreateTablet for tablet e32567235f7e49b6958a822e4836409b (DEFAULT_TABLE table=default.Table_1d3219f0687d4cb595ce0dae72cfb41a [id=bc65fa5b7dd94ccfaa3d89164a946988]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.984490 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 36817f0405134207ac05a4e154a6887b (DEFAULT_TABLE table=default.Table_00c020c7924a4724baf7a9d36a9971b7 [id=d09c17e66643456fa7d5a8f2984e4c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.984509 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 97166d0050544ba6b56a329e1a452e66 (DEFAULT_TABLE table=default.Table_48c282098fff4f29af36cc85c62ec34a [id=40c9e39d4c9541aba36037b89318581f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.984856 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36817f0405134207ac05a4e154a6887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.985064 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97166d0050544ba6b56a329e1a452e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.987861 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet f407b76780974cf7b3635fa1bf1585ec (table default.Table_6859465fc7f4440eaf4c387bc97b02ad [id=32d713e3cdee403cb9193769815a07fa]). Expected version 2 got 0
I20250109 20:53:11.988086 448 catalog_manager.cc:5526] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 09833ea46b7141d58c3899dace69e49f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "09833ea46b7141d58c3899dace69e49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: UNKNOWN } } }
I20250109 20:53:11.988399 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 74f4bdda16554d9d81de5ca5c5205f4c (table default.Table_d2d8f346b1d2494cb588d606237080a0 [id=dfdbf927d4ff4f329975148ef22d8940]). Expected version 1 got 0
I20250109 20:53:11.988584 448 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 2420b285905342b3820f9549c8f0c4ad (table default.Table_c9b68cce9a8544a38a418f69637719b6 [id=ec17429d6df548118b1e53e77bac80c0]). Expected version 3 got 0
I20250109 20:53:11.991741 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 3e45125e7ca14225b67c0cf92f76b312
I20250109 20:53:11.992632 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e45125e7ca14225b67c0cf92f76b312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.993229 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e45125e7ca14225b67c0cf92f76b312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:11.995857 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b795e3740bf549eb86f766da5b3b9a4c (DEFAULT_TABLE table=default.Table_3a973775fce94f7493d3ca21e4698bf0 [id=cd5618287f034540a02dc01f7e51c15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.996042 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_05ab5bd90c8d4f3e97a59539dbeffdc8" } new_table_name: "default.Table_0ba119da18824d5eb0304834dc396a7d" modify_external_catalogs: true
I20250109 20:53:11.996296 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b795e3740bf549eb86f766da5b3b9a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:11.998099 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet a0aae7613aaf4248b4b8ada6e60fe8ea
I20250109 20:53:11.998564 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 58f3afa4595a41baa77723f14d6a4944
I20250109 20:53:11.999616 744 tablet_service.cc:1514] Processing DeleteTablet for tablet a0aae7613aaf4248b4b8ada6e60fe8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:11.999810 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f3afa4595a41baa77723f14d6a4944 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.000000 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 014ad388a4684138980929e5c3d8e4e2 (DEFAULT_TABLE table=default.Table_bd42ddc2011e4030b0063d864e6972db [id=61764b73f41f4d659e17a0ca12e967b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.000252 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f3afa4595a41baa77723f14d6a4944 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.000270 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet a0aae7613aaf4248b4b8ada6e60fe8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.000957 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 014ad388a4684138980929e5c3d8e4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.005061 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_31cc0ffd3c1c44f3876b4793eae7b026" } new_table_name: "default.Table_322d0b49ab5d4d5cb54c396a942a316e" modify_external_catalogs: true
I20250109 20:53:12.005795 1166 ts_tablet_manager.cc:1918] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.005963 1166 log.cc:1198] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/418e1bd414914b21a233c3099e2057da
I20250109 20:53:12.006217 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_ae206aa8feab40e897d223b97a6c164b" } new_table_name: "default.Table_12fe955f8b2748209d68df1d37c0fe42" modify_external_catalogs: true
I20250109 20:53:12.006382 1166 ts_tablet_manager.cc:1939] T 418e1bd414914b21a233c3099e2057da P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:12.007277 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_201a598c2da7487bbb7cd29c9a0d9ae1" } new_table_name: "default.Table_a6d074b4b34246ec8f5680659177d145" modify_external_catalogs: true
I20250109 20:53:12.007454 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 418e1bd414914b21a233c3099e2057da (table default.Table_5377c27f242f4aaf8ef9d0281640a856 [id=7521b80e658f4aaf9761df625056858c]) successfully deleted
W20250109 20:53:12.008042 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 3e45125e7ca14225b67c0cf92f76b312 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e45125e7ca14225b67c0cf92f76b312 already in progress: creating tablet
I20250109 20:53:12.008294 1176 tablet_bootstrap.cc:492] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:12.008411 1176 ts_tablet_manager.cc:1397] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20250109 20:53:12.008948 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 99facf2f6a7c4afc96ff0e06b58da937 (DEFAULT_TABLE table=default.Table_025053d76dd84c2a9a5d320fa3a085d9 [id=ee89624f7ec5426f92789da48e8dde27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.009192 1176 raft_consensus.cc:357] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.009371 1176 raft_consensus.cc:383] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.009435 1176 raft_consensus.cc:738] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.009465 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99facf2f6a7c4afc96ff0e06b58da937. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.011152 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9b490cee97c442edac131ba0a23a3e75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.011551 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b490cee97c442edac131ba0a23a3e75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.013962 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d758bb29a9294e9481bb15ca4b4429b0
I20250109 20:53:11.978641 740 tablet_service.cc:1467] Processing CreateTablet for tablet 014ad388a4684138980929e5c3d8e4e2 (DEFAULT_TABLE table=default.Table_bd42ddc2011e4030b0063d864e6972db [id=61764b73f41f4d659e17a0ca12e967b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.978741 746 tablet_service.cc:1467] Processing CreateTablet for tablet b795e3740bf549eb86f766da5b3b9a4c (DEFAULT_TABLE table=default.Table_3a973775fce94f7493d3ca21e4698bf0 [id=cd5618287f034540a02dc01f7e51c15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.014282 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet c50325c8a0e34acaa6e2912d3923d43f
I20250109 20:53:11.979147 749 tablet_service.cc:1467] Processing CreateTablet for tablet 36817f0405134207ac05a4e154a6887b (DEFAULT_TABLE table=default.Table_00c020c7924a4724baf7a9d36a9971b7 [id=d09c17e66643456fa7d5a8f2984e4c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:11.979791 745 tablet_service.cc:1467] Processing CreateTablet for tablet 97166d0050544ba6b56a329e1a452e66 (DEFAULT_TABLE table=default.Table_48c282098fff4f29af36cc85c62ec34a [id=40c9e39d4c9541aba36037b89318581f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.014532 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_81d6c3babee64540bb908d606439d383"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.014793 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bca6602ca0534bf8ab4b47d6e287ee2e consensus state has no leader
W20250109 20:53:12.014907 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81d6c3babee64540bb908d606439d383 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:11.980055 742 tablet_service.cc:1467] Processing CreateTablet for tablet 81440079bca14ec08ed6716b786a4f73 (DEFAULT_TABLE table=default.Table_8252859a8fcd44b0984a8c054e3c0416 [id=47f6f9f7e158477aa8a86da4340ee906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.015280 755 tablet_service.cc:1514] Processing DeleteTablet for tablet d758bb29a9294e9481bb15ca4b4429b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.015481 744 tablet_service.cc:1514] Processing DeleteTablet for tablet c50325c8a0e34acaa6e2912d3923d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.017136 453 catalog_manager.cc:5526] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5fcc647ed479491b88e5ffacf2927a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } health_report { overall_health: HEALTHY } } }
I20250109 20:53:12.017495 453 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet c0860554cac64b8882e39fa62462b1aa (table default.Table_f74ab9f194134e6ba390983fba09d20a [id=f36b9d8c432d4f108296cd5576832573]). Expected version 2 got 0
I20250109 20:53:12.017604 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet c50325c8a0e34acaa6e2912d3923d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.018472 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d758bb29a9294e9481bb15ca4b4429b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.023589 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_b37028a3586243068596a58b468addf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:12.023614 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9638d982d8af46ffa56e6f5cd62385b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.024056 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9638d982d8af46ffa56e6f5cd62385b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.023995 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_df11090bd12744ba9e450b538f77c5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.024153 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 58f3afa4595a41baa77723f14d6a4944 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58f3afa4595a41baa77723f14d6a4944 already in progress: creating tablet
W20250109 20:53:12.024322 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df11090bd12744ba9e450b538f77c5e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:12.024631 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet a0aae7613aaf4248b4b8ada6e60fe8ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0aae7613aaf4248b4b8ada6e60fe8ea already in progress: creating tablet
W20250109 20:53:12.025166 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet c50325c8a0e34acaa6e2912d3923d43f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c50325c8a0e34acaa6e2912d3923d43f already in progress: creating tablet
W20250109 20:53:12.025303 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b37028a3586243068596a58b468addf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.025859 1160 tablet_replica.cc:331] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.026093 1160 raft_consensus.cc:2238] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.026289 1160 raft_consensus.cc:2267] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.026906 1160 ts_tablet_manager.cc:1905] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:12.028815 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93466e0ad0b444ada255c80494f1967e consensus state has no leader
W20250109 20:53:12.035419 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d758bb29a9294e9481bb15ca4b4429b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d758bb29a9294e9481bb15ca4b4429b0 already in progress: creating tablet
I20250109 20:53:12.036154 1165 tablet_bootstrap.cc:492] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.036295 1165 ts_tablet_manager.cc:1397] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.109s user 0.001s sys 0.016s
I20250109 20:53:12.037216 1165 raft_consensus.cc:357] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.037391 1165 raft_consensus.cc:383] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.037437 1165 raft_consensus.cc:738] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.037604 1165 consensus_queue.cc:260] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.038028 1165 ts_tablet_manager.cc:1428] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:12.038254 1165 tablet_bootstrap.cc:492] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.039770 1165 tablet_bootstrap.cc:654] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.041231 1158 tablet_bootstrap.cc:492] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.041390 1158 ts_tablet_manager.cc:1397] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.154s user 0.014s sys 0.016s
I20250109 20:53:12.042121 1158 raft_consensus.cc:357] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.042312 1158 raft_consensus.cc:383] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.042376 1158 raft_consensus.cc:738] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.042570 1158 consensus_queue.cc:260] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.043030 1158 ts_tablet_manager.cc:1428] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:12.043690 1158 tablet_bootstrap.cc:492] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.044308 1160 ts_tablet_manager.cc:1918] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.044396 1160 log.cc:1198] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/2769e5906c604bd5b14df79db765ec61
I20250109 20:53:12.045450 1158 tablet_bootstrap.cc:654] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.049242 1176 consensus_queue.cc:260] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.049417 744 tablet_service.cc:1467] Processing CreateTablet for tablet 990189779f084e87942736ea36a14e4b (DEFAULT_TABLE table=default.Table_1d1de589c7d742b2a63f6216b5655247 [id=1209c5dc31e2461e943f057eafeefcb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.049496 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 990189779f084e87942736ea36a14e4b (DEFAULT_TABLE table=default.Table_1d1de589c7d742b2a63f6216b5655247 [id=1209c5dc31e2461e943f057eafeefcb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.049923 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97166d0050544ba6b56a329e1a452e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.050123 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 990189779f084e87942736ea36a14e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.050307 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36817f0405134207ac05a4e154a6887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.052872 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81440079bca14ec08ed6716b786a4f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.053256 1165 tablet_bootstrap.cc:492] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.053359 1165 ts_tablet_manager.cc:1397] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250109 20:53:12.053951 1165 raft_consensus.cc:357] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.054119 1165 raft_consensus.cc:383] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.054158 1165 raft_consensus.cc:738] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.054335 1165 consensus_queue.cc:260] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.054787 1165 ts_tablet_manager.cc:1428] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:12.055025 1165 tablet_bootstrap.cc:492] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.055524 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b795e3740bf549eb86f766da5b3b9a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.056461 1165 tablet_bootstrap.cc:654] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.060685 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32567235f7e49b6958a822e4836409b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.062124 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 990189779f084e87942736ea36a14e4b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:12.064347 2887 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:53:11.970306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:41583 (local address 127.0.104.2:33963)
0109 20:53:12.064087 (+ 93781us) negotiation.cc:107] Waiting for socket to connect
0109 20:53:12.064175 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":93659,"threads_started":1}
I20250109 20:53:12.065279 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99facf2f6a7c4afc96ff0e06b58da937. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:12.067031 2884 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:53:11.969476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:41583 (local address 127.0.104.3:50945)
0109 20:53:12.066729 (+ 97253us) negotiation.cc:107] Waiting for socket to connect
0109 20:53:12.066825 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":97140,"threads_started":1}
W20250109 20:53:12.067366 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9615f0d1bb9d46ba8e855fb5819eb5a6 consensus state has no leader
I20250109 20:53:12.067469 1158 tablet_bootstrap.cc:492] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.067625 1158 ts_tablet_manager.cc:1397] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
W20250109 20:53:12.067770 2885 negotiation.cc:337] Failed RPC negotiation. Trace:
0109 20:53:11.969779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:41583 (local address 127.0.104.1:47103)
0109 20:53:12.067442 (+ 97663us) negotiation.cc:107] Waiting for socket to connect
0109 20:53:12.067564 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":97571,"threads_started":1}
W20250109 20:53:12.067922 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41583, user_credentials={real_user=slave}} blocked reactor thread for 99668.7us
I20250109 20:53:12.068446 1158 raft_consensus.cc:357] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.068645 1158 raft_consensus.cc:383] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.068774 1158 raft_consensus.cc:738] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.068693 898 tablet_service.cc:1467] Processing CreateTablet for tablet 99facf2f6a7c4afc96ff0e06b58da937 (DEFAULT_TABLE table=default.Table_025053d76dd84c2a9a5d320fa3a085d9 [id=ee89624f7ec5426f92789da48e8dde27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.069018 1158 consensus_queue.cc:260] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.069376 885 tablet_service.cc:1467] Processing CreateTablet for tablet 014ad388a4684138980929e5c3d8e4e2 (DEFAULT_TABLE table=default.Table_bd42ddc2011e4030b0063d864e6972db [id=61764b73f41f4d659e17a0ca12e967b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.069376 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99facf2f6a7c4afc96ff0e06b58da937. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.069840 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 014ad388a4684138980929e5c3d8e4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.069788 892 tablet_service.cc:1467] Processing CreateTablet for tablet 81440079bca14ec08ed6716b786a4f73 (DEFAULT_TABLE table=default.Table_8252859a8fcd44b0984a8c054e3c0416 [id=47f6f9f7e158477aa8a86da4340ee906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.070115 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81440079bca14ec08ed6716b786a4f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.072803 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 014ad388a4684138980929e5c3d8e4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.076244 892 tablet_service.cc:1467] Processing CreateTablet for tablet e32567235f7e49b6958a822e4836409b (DEFAULT_TABLE table=default.Table_1d3219f0687d4cb595ce0dae72cfb41a [id=bc65fa5b7dd94ccfaa3d89164a946988]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.076763 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e32567235f7e49b6958a822e4836409b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:12.077585 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85a8bfb21cb44e14bdacdf8b8208a116 consensus state has no leader
I20250109 20:53:12.080482 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_e1f4f1911eab439d93ae6e3919fe821b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.081014 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1f4f1911eab439d93ae6e3919fe821b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.082635 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5609eaf86d4e8ea3d3da41243c927f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.082962 895 tablet_service.cc:1467] Processing CreateTablet for tablet b795e3740bf549eb86f766da5b3b9a4c (DEFAULT_TABLE table=default.Table_3a973775fce94f7493d3ca21e4698bf0 [id=cd5618287f034540a02dc01f7e51c15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.083245 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b795e3740bf549eb86f766da5b3b9a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.088272 1158 ts_tablet_manager.cc:1428] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250109 20:53:12.088749 895 tablet_service.cc:1467] Processing CreateTablet for tablet 36817f0405134207ac05a4e154a6887b (DEFAULT_TABLE table=default.Table_00c020c7924a4724baf7a9d36a9971b7 [id=d09c17e66643456fa7d5a8f2984e4c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.089020 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36817f0405134207ac05a4e154a6887b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.091886 1158 tablet_bootstrap.cc:492] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.093473 1158 tablet_bootstrap.cc:654] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.094405 1165 tablet_bootstrap.cc:492] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.094537 1165 ts_tablet_manager.cc:1397] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20250109 20:53:12.094767 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e45125e7ca14225b67c0cf92f76b312 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.095224 885 tablet_service.cc:1514] Processing DeleteTablet for tablet a0aae7613aaf4248b4b8ada6e60fe8ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.095297 1165 raft_consensus.cc:357] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.095454 1165 raft_consensus.cc:383] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.095505 1165 raft_consensus.cc:738] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.095631 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f3afa4595a41baa77723f14d6a4944 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.095716 1165 consensus_queue.cc:260] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.096066 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d758bb29a9294e9481bb15ca4b4429b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.096251 1165 ts_tablet_manager.cc:1428] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.096470 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c50325c8a0e34acaa6e2912d3923d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:11 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.096997 885 tablet_service.cc:1467] Processing CreateTablet for tablet 990189779f084e87942736ea36a14e4b (DEFAULT_TABLE table=default.Table_1d1de589c7d742b2a63f6216b5655247 [id=1209c5dc31e2461e943f057eafeefcb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.097554 898 tablet_service.cc:1467] Processing CreateTablet for tablet a06a0afdfe2f435884c3e4f1e4b19fd2 (DEFAULT_TABLE table=default.Table_8594ac97908c49dba72c916b11b44850 [id=b6f61682225e4498951d79210fa5abef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.097630 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 990189779f084e87942736ea36a14e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.097795 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a06a0afdfe2f435884c3e4f1e4b19fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.098037 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814361a1e854451a0ee7469cc6d447b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:49 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.096489 1165 tablet_bootstrap.cc:492] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.103036 895 tablet_service.cc:1467] Processing CreateTablet for tablet 97166d0050544ba6b56a329e1a452e66 (DEFAULT_TABLE table=default.Table_48c282098fff4f29af36cc85c62ec34a [id=40c9e39d4c9541aba36037b89318581f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.104302 1165 tablet_bootstrap.cc:654] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.105281 1176 ts_tablet_manager.cc:1428] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.097s user 0.002s sys 0.000s
W20250109 20:53:12.106570 989 leader_election.cc:343] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.106699 898 tablet_service.cc:1514] Processing DeleteTablet for tablet bd62a3ae64a143998bb1172b3a97c69f with delete_type TABLET_DATA_DELETED (replaced by tablet 575f3cb9f61143ab949fa85f47edcafb) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.106712 989 leader_election.cc:304] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.106945 2822 raft_consensus.cc:2743] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.110324 740 tablet_service.cc:1467] Processing CreateTablet for tablet a06a0afdfe2f435884c3e4f1e4b19fd2 (DEFAULT_TABLE table=default.Table_8594ac97908c49dba72c916b11b44850 [id=b6f61682225e4498951d79210fa5abef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.110430 1165 tablet_bootstrap.cc:492] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.110538 1165 ts_tablet_manager.cc:1397] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250109 20:53:12.110814 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a06a0afdfe2f435884c3e4f1e4b19fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.111176 1165 raft_consensus.cc:357] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.111349 1165 raft_consensus.cc:383] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.111415 1165 raft_consensus.cc:738] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.111606 1165 consensus_queue.cc:260] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.112020 1165 ts_tablet_manager.cc:1428] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:12.112221 1165 tablet_bootstrap.cc:492] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.112838 1158 tablet_bootstrap.cc:492] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.112958 1158 ts_tablet_manager.cc:1397] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250109 20:53:12.113606 1158 raft_consensus.cc:357] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.113777 1158 raft_consensus.cc:383] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.113850 1158 raft_consensus.cc:738] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.114042 1158 consensus_queue.cc:260] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.044677 1160 ts_tablet_manager.cc:1939] T 2769e5906c604bd5b14df79db765ec61 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.111372 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97166d0050544ba6b56a329e1a452e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.115504 1160 tablet_replica.cc:331] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.115674 1160 raft_consensus.cc:2238] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.115809 1160 raft_consensus.cc:2267] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.115998 1031 tablet_service.cc:1467] Processing CreateTablet for tablet a06a0afdfe2f435884c3e4f1e4b19fd2 (DEFAULT_TABLE table=default.Table_8594ac97908c49dba72c916b11b44850 [id=b6f61682225e4498951d79210fa5abef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.116473 1160 ts_tablet_manager.cc:1905] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.121516 1165 tablet_bootstrap.cc:654] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.121493 2695 raft_consensus.cc:491] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.121618 2695 raft_consensus.cc:513] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.122224 2695 leader_election.cc:290] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.122586 1158 ts_tablet_manager.cc:1428] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250109 20:53:12.123003 1158 tablet_bootstrap.cc:492] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.122924 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_7c9988bb451f47b3a543402c197eadea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.123447 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c9988bb451f47b3a543402c197eadea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.124485 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3008c672bc544000b24b75ae366d9fb9" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.124938 712 leader_election.cc:343] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.125144 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6dfcd0e67d70483c8c0f0fcf161f8789"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.125558 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dfcd0e67d70483c8c0f0fcf161f8789 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.126156 1176 tablet_bootstrap.cc:492] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:12.126645 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_45791cc3612f483986b22cd9349edda0" } new_table_name: "default.Table_8b17dc95b5594a29a04e107d60275ef0" modify_external_catalogs: true
I20250109 20:53:12.126968 1158 tablet_bootstrap.cc:654] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.128398 2822 raft_consensus.cc:491] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.128530 2822 raft_consensus.cc:513] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.129176 2822 leader_election.cc:290] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:12.130436 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade6b4ea4ca44f3eb88d3911b0bbb70b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:12.130928 990 leader_election.cc:343] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.131934 752 tablet_service.cc:1467] Processing CreateTablet for tablet a6434cbc16eb47879089491e19d8bff6 (DEFAULT_TABLE table=default.Table_4ab544cd23024498b1983ef3bd7042d5 [id=e464682be8c94bd2854daefd98732810]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.132288 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6434cbc16eb47879089491e19d8bff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.133630 1165 tablet_bootstrap.cc:492] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.133745 1165 ts_tablet_manager.cc:1397] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.005s
I20250109 20:53:12.134609 1165 raft_consensus.cc:357] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.134788 1165 raft_consensus.cc:383] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.134841 1165 raft_consensus.cc:738] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.135041 1165 consensus_queue.cc:260] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.135526 1165 ts_tablet_manager.cc:1428] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.135752 1165 tablet_bootstrap.cc:492] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.135731 2695 raft_consensus.cc:491] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.135850 2695 raft_consensus.cc:513] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.135905 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_75b27c586c024babaaaef44859604050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.136387 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75b27c586c024babaaaef44859604050 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.136477 2695 leader_election.cc:290] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.137440 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c83e709a8d4e078e67af58ec51beac" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.137854 712 leader_election.cc:343] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.138221 1165 tablet_bootstrap.cc:654] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.139546 1158 tablet_bootstrap.cc:492] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.139663 1176 tablet_bootstrap.cc:654] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.139667 1158 ts_tablet_manager.cc:1397] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20250109 20:53:12.140303 1158 raft_consensus.cc:357] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.140480 1158 raft_consensus.cc:383] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.140544 1158 raft_consensus.cc:738] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.140748 1158 consensus_queue.cc:260] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.141413 1158 ts_tablet_manager.cc:1428] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:12.141651 1158 tablet_bootstrap.cc:492] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.143726 1158 tablet_bootstrap.cc:654] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.144500 1160 ts_tablet_manager.cc:1918] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.144609 1160 log.cc:1198] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/9d7587432d7345e4a225fc7d5c968c27
I20250109 20:53:12.144927 1160 ts_tablet_manager.cc:1939] T 9d7587432d7345e4a225fc7d5c968c27 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.147724 1165 tablet_bootstrap.cc:492] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.147814 1165 ts_tablet_manager.cc:1397] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20250109 20:53:12.148401 1165 raft_consensus.cc:357] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.148564 1165 raft_consensus.cc:383] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.148604 1165 raft_consensus.cc:738] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.148967 1165 consensus_queue.cc:260] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.149547 1165 ts_tablet_manager.cc:1428] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.150662 1158 tablet_bootstrap.cc:492] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.150820 1158 ts_tablet_manager.cc:1397] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20250109 20:53:12.150985 1157 ts_tablet_manager.cc:1918] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.151142 1157 log.cc:1198] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/3293f49190684b72a5c41031e621bd36
I20250109 20:53:12.151484 1157 ts_tablet_manager.cc:1939] T 3293f49190684b72a5c41031e621bd36 P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:12.151804 1158 raft_consensus.cc:357] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.152057 1158 raft_consensus.cc:383] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.152143 1158 raft_consensus.cc:738] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.152422 1158 consensus_queue.cc:260] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.152607 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 3293f49190684b72a5c41031e621bd36 (table default.Table_2c68d926e1b8418aa0339da1c4cb17cf [id=83feb9b334c14673b257b605634f7bd8]) successfully deleted
I20250109 20:53:12.152957 1157 tablet_replica.cc:331] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:12.153993 1034 tablet_service.cc:1467] Processing CreateTablet for tablet a6434cbc16eb47879089491e19d8bff6 (DEFAULT_TABLE table=default.Table_4ab544cd23024498b1983ef3bd7042d5 [id=e464682be8c94bd2854daefd98732810]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.155027 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f8352c4ff1784f82a29b859fbe422481 (DEFAULT_TABLE table=default.Table_d9affec4411a4ff4aeb64926df8c7e47 [id=7989a3efc84f4f02aa5d71adb7fa20b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.156596 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca8bc2348954ad9bcea555665928b1b (DEFAULT_TABLE table=default.Table_0c3b555d74764dc09f834d3ffb88e038 [id=ddd9474d31d0479abbb8ad68ff9bf50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.158471 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 787aa70978c146529d46cbb493d417de (DEFAULT_TABLE table=default.Table_faeda3c4ba4943eaaf3ecd5dadbcfe55 [id=a0806524f7dd475dbcc96fde612bce58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.160634 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c52c681f39594d6d89a48cda7a0eda6f (DEFAULT_TABLE table=default.Table_ef211780e3a44c84b59dca475eb0006e [id=450e4a3e528e414b96a463ccdb9dbc6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.165131 2818 raft_consensus.cc:491] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.165256 2818 raft_consensus.cc:513] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.165817 2818 leader_election.cc:290] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.166999 854 leader_election.cc:343] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: bd62a3ae64a143998bb1172b3a97c69f
I20250109 20:53:12.167315 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade6b4ea4ca44f3eb88d3911b0bbb70b" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.167701 2822 raft_consensus.cc:491] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:53:12.167920 989 leader_election.cc:343] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.167834 2822 raft_consensus.cc:513] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.168023 989 leader_election.cc:304] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.168259 2826 raft_consensus.cc:2743] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.168504 2822 leader_election.cc:290] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:12.169478 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f53cef0adbb47d4840e4c11a6d0c8ac" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.177386 854 leader_election.cc:343] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: bd62a3ae64a143998bb1172b3a97c69f
I20250109 20:53:12.177506 854 leader_election.cc:304] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.177848 2818 raft_consensus.cc:2743] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.177922 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6434cbc16eb47879089491e19d8bff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.178115 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca8bc2348954ad9bcea555665928b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.180893 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 787aa70978c146529d46cbb493d417de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.183585 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a06a0afdfe2f435884c3e4f1e4b19fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.184935 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52c681f39594d6d89a48cda7a0eda6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.186910 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: default.Table_bc4981434b6743fc93063e84ec2ed645 [id=28723b313bd34951826646276c761571] alter complete (version 3)
I20250109 20:53:12.188920 1160 tablet_replica.cc:331] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.189095 1160 raft_consensus.cc:2238] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.189226 1160 raft_consensus.cc:2267] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.189563 1165 tablet_bootstrap.cc:492] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.190452 2818 raft_consensus.cc:491] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.190565 2818 raft_consensus.cc:513] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.190820 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8352c4ff1784f82a29b859fbe422481. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.191125 1165 tablet_bootstrap.cc:654] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.191161 2818 leader_election.cc:290] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.194013 854 leader_election.cc:343] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: fce062dc0ac144a19f8fa3eef4a099b4
W20250109 20:53:12.194182 854 leader_election.cc:343] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: fce062dc0ac144a19f8fa3eef4a099b4
I20250109 20:53:12.194252 854 leader_election.cc:304] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.194965 2818 raft_consensus.cc:2743] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.196177 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 65570e42a9bd45bcba0a82abd12a9356 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:53 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.196339 1160 ts_tablet_manager.cc:1905] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.196375 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet ade6b4ea4ca44f3eb88d3911b0bbb70b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:08 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.196589 1166 tablet_replica.cc:331] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:12.196882 1166 raft_consensus.cc:2238] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.197083 1166 raft_consensus.cc:2267] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.197937 1166 ts_tablet_manager.cc:1905] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.198272 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_188ed9bb91204948b8552c0b799b00d2" } modify_external_catalogs: true
I20250109 20:53:12.198436 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_188ed9bb91204948b8552c0b799b00d2" } modify_external_catalogs: true
I20250109 20:53:12.201545 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5609eaf86d4e8ea3d3da41243c927f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:52 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.205701 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dadeccdf43324b11a72aad7f2d19b825" } modify_external_catalogs: true
I20250109 20:53:12.205896 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_dadeccdf43324b11a72aad7f2d19b825" } modify_external_catalogs: true
I20250109 20:53:12.206154 2822 raft_consensus.cc:491] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.206290 2822 raft_consensus.cc:513] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.206934 2822 leader_election.cc:290] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:12.217208 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c717e446858400789dd0e84b4fb60cd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.217716 1160 ts_tablet_manager.cc:1918] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.217862 1160 log.cc:1198] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4870a667cd3e410590923c9e2497af54
I20250109 20:53:12.218226 1160 ts_tablet_manager.cc:1939] T 4870a667cd3e410590923c9e2497af54 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.153209 1158 ts_tablet_manager.cc:1428] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.155674 749 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca8bc2348954ad9bcea555665928b1b (DEFAULT_TABLE table=default.Table_0c3b555d74764dc09f834d3ffb88e038 [id=ddd9474d31d0479abbb8ad68ff9bf50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.157697 746 tablet_service.cc:1467] Processing CreateTablet for tablet 787aa70978c146529d46cbb493d417de (DEFAULT_TABLE table=default.Table_faeda3c4ba4943eaaf3ecd5dadbcfe55 [id=a0806524f7dd475dbcc96fde612bce58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.159919 739 tablet_service.cc:1467] Processing CreateTablet for tablet c52c681f39594d6d89a48cda7a0eda6f (DEFAULT_TABLE table=default.Table_ef211780e3a44c84b59dca475eb0006e [id=450e4a3e528e414b96a463ccdb9dbc6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.219571 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5fef44f0700046e0ba377025bb4d4c78" } modify_external_catalogs: true
I20250109 20:53:12.161818 742 tablet_service.cc:1467] Processing CreateTablet for tablet 6893f768a63b4abf8224621ccfb66a57 (DEFAULT_TABLE table=default.Table_507ede4faf0d4c83b5844d0247715180 [id=0a31e04681564cfb9224780341f0c47b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.161936 752 tablet_service.cc:1467] Processing CreateTablet for tablet f8352c4ff1784f82a29b859fbe422481 (DEFAULT_TABLE table=default.Table_d9affec4411a4ff4aeb64926df8c7e47 [id=7989a3efc84f4f02aa5d71adb7fa20b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.169039 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f53cef0adbb47d4840e4c11a6d0c8ac" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:12.219714 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5fef44f0700046e0ba377025bb4d4c78" } modify_external_catalogs: true
I20250109 20:53:12.220022 895 tablet_service.cc:1514] Processing DeleteTablet for tablet ea16fcc30f2c4b2eac53f20fcebd043f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:50 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.188449 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 3008c672bc544000b24b75ae366d9fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:57 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.192044 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9170b3a6b9bb446b8a474ae4df249409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:51 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.195463 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2fcddba2ac4d82b2d25bfc64a5967e with delete_type TABLET_DATA_DELETED (replaced by tablet 37ee437300414eccba9926e824f72f27) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.207504 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c717e446858400789dd0e84b4fb60cd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:12.220548 778 raft_consensus.cc:2463] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
W20250109 20:53:12.216544 713 leader_election.cc:343] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 3008c672bc544000b24b75ae366d9fb9
I20250109 20:53:12.220914 713 leader_election.cc:304] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.221000 757 tablet_service.cc:1467] Processing CreateTablet for tablet b7fee30e616b497f99d91ee199fa950a (DEFAULT_TABLE table=default.Table_0d846b274cdc42229ed64d25124b2560 [id=caefc0759f6042d8845c513375fcf11c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:12.221395 713 consensus_peers.cc:487] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f -> Peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Couldn't send request to peer 5fcc647ed479491b88e5ffacf2927a1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c5e9b31b27949c2948cb64b6124ee72. This is attempt 1: this message will repeat every 5th retry.
W20250109 20:53:12.221642 713 leader_election.cc:343] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: f2c83e709a8d4e078e67af58ec51beac
I20250109 20:53:12.221737 713 leader_election.cc:304] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.221984 2692 raft_consensus.cc:2743] T 3008c672bc544000b24b75ae366d9fb9 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.222076 895 tablet_service.cc:1514] Processing DeleteTablet for tablet de8ea5ea7fa44866b6d7160466f5a896 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:48 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.222379 2692 raft_consensus.cc:2743] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.222529 2692 raft_consensus.cc:491] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.222651 2692 raft_consensus.cc:513] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.223232 2692 leader_election.cc:290] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.223326 744 tablet_service.cc:1467] Processing CreateTablet for tablet 37efddac3f8147e9a27ac0671f390d18 (DEFAULT_TABLE table=default.Table_e86358bf98744df4a2432305fcffde2b [id=e91423edbc354a93bafdfd83a4763f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.223482 895 tablet_service.cc:1467] Processing CreateTablet for tablet a6434cbc16eb47879089491e19d8bff6 (DEFAULT_TABLE table=default.Table_4ab544cd23024498b1983ef3bd7042d5 [id=e464682be8c94bd2854daefd98732810]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.223730 1157 raft_consensus.cc:2238] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.223883 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6434cbc16eb47879089491e19d8bff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.223922 1157 raft_consensus.cc:2267] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.226737 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9170b3a6b9bb446b8a474ae4df249409" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.227254 712 leader_election.cc:343] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.222440 1160 tablet_replica.cc:331] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.229022 885 tablet_service.cc:1467] Processing CreateTablet for tablet f8352c4ff1784f82a29b859fbe422481 (DEFAULT_TABLE table=default.Table_d9affec4411a4ff4aeb64926df8c7e47 [id=7989a3efc84f4f02aa5d71adb7fa20b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.229514 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8352c4ff1784f82a29b859fbe422481. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.233397 1158 tablet_bootstrap.cc:492] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.234086 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca8bc2348954ad9bcea555665928b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.234912 1158 tablet_bootstrap.cc:654] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.234856 740 tablet_service.cc:1467] Processing CreateTablet for tablet 7889352df8064db7ad5180ac075397ff (DEFAULT_TABLE table=default.Table_85817f72c90941a2af0506a8759f34c3 [id=fd57b15ff31745ecafe1b05ab096c975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.235180 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7889352df8064db7ad5180ac075397ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.235810 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 787aa70978c146529d46cbb493d417de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.235983 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52c681f39594d6d89a48cda7a0eda6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.236868 1160 raft_consensus.cc:2238] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.237000 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fee30e616b497f99d91ee199fa950a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.237090 1160 raft_consensus.cc:2267] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.237860 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca8bc2348954ad9bcea555665928b1b (DEFAULT_TABLE table=default.Table_0c3b555d74764dc09f834d3ffb88e038 [id=ddd9474d31d0479abbb8ad68ff9bf50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.238301 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca8bc2348954ad9bcea555665928b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.240945 2818 raft_consensus.cc:491] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.241080 2818 raft_consensus.cc:513] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.241696 2818 leader_election.cc:290] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.241662 895 tablet_service.cc:1467] Processing CreateTablet for tablet 787aa70978c146529d46cbb493d417de (DEFAULT_TABLE table=default.Table_faeda3c4ba4943eaaf3ecd5dadbcfe55 [id=a0806524f7dd475dbcc96fde612bce58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.241937 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8352c4ff1784f82a29b859fbe422481. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.242033 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 787aa70978c146529d46cbb493d417de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.247556 1158 tablet_bootstrap.cc:492] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.249353 892 tablet_service.cc:1467] Processing CreateTablet for tablet c52c681f39594d6d89a48cda7a0eda6f (DEFAULT_TABLE table=default.Table_ef211780e3a44c84b59dca475eb0006e [id=450e4a3e528e414b96a463ccdb9dbc6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.249755 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52c681f39594d6d89a48cda7a0eda6f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:12.249805 854 leader_election.cc:343] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: de8ea5ea7fa44866b6d7160466f5a896
I20250109 20:53:12.247692 1158 ts_tablet_manager.cc:1397] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20250109 20:53:12.253247 1158 raft_consensus.cc:357] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.253419 1158 raft_consensus.cc:383] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.253458 1158 raft_consensus.cc:738] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.253623 1158 consensus_queue.cc:260] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.254040 1158 ts_tablet_manager.cc:1428] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20250109 20:53:12.254442 1158 tablet_bootstrap.cc:492] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.254482 2695 raft_consensus.cc:491] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.254597 2695 raft_consensus.cc:513] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.255095 2695 leader_election.cc:290] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.255399 892 tablet_service.cc:1467] Processing CreateTablet for tablet 6893f768a63b4abf8224621ccfb66a57 (DEFAULT_TABLE table=default.Table_507ede4faf0d4c83b5844d0247715180 [id=0a31e04681564cfb9224780341f0c47b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.255534 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eaa5bbfaf4c1c9d8cd1b1ead5f632" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.255704 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6893f768a63b4abf8224621ccfb66a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.256156 1158 tablet_bootstrap.cc:654] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.256297 712 leader_election.cc:343] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.259522 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 0019a01afbcc452a9a9374c92dc131f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a4b7e008a474978b91426f63dbd96ee) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:12.259721 440 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 -> Peer 36a29ab2002d4eb389cd11778988e1e9 (127.0.104.60:41583): Couldn't send request to peer 36a29ab2002d4eb389cd11778988e1e9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:12.260233 898 tablet_service.cc:1467] Processing CreateTablet for tablet b7fee30e616b497f99d91ee199fa950a (DEFAULT_TABLE table=default.Table_0d846b274cdc42229ed64d25124b2560 [id=caefc0759f6042d8845c513375fcf11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.260510 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fee30e616b497f99d91ee199fa950a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.260483 752 tablet_service.cc:1467] Processing CreateTablet for tablet cb28bf037752421295e3e44785363882 (DEFAULT_TABLE table=default.Table_9d5da32f48064377ad7a7942e7122a8f [id=c0c7b6f20cd34261adf9e7dc9aaa0d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.260803 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb28bf037752421295e3e44785363882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.267009 1160 ts_tablet_manager.cc:1905] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.267043 2695 raft_consensus.cc:491] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.267154 2695 raft_consensus.cc:513] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.267578 2805 raft_consensus.cc:491] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.267676 2695 leader_election.cc:290] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.267673 2805 raft_consensus.cc:513] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.268154 2805 leader_election.cc:290] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.268291 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f407b76780974cf7b3635fa1bf1585ec" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.268489 2805 raft_consensus.cc:491] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.268591 2805 raft_consensus.cc:513] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
W20250109 20:53:12.268921 712 leader_election.cc:343] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.269157 2805 leader_election.cc:290] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.269354 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b285905342b3820f9549c8f0c4ad" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.269824 712 leader_election.cc:343] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.270159 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374df8c6d1424697a3b3922f777e9870" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.270539 712 leader_election.cc:343] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.270692 2805 raft_consensus.cc:491] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.270792 2805 raft_consensus.cc:513] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.271294 2805 leader_election.cc:290] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.271870 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45cd5ffe7040e385d8cdfb97ed011a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.272296 712 leader_election.cc:343] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.272442 1160 ts_tablet_manager.cc:1918] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.272529 1160 log.cc:1198] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/fce062dc0ac144a19f8fa3eef4a099b4
I20250109 20:53:12.268601 1158 tablet_bootstrap.cc:492] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.272868 1158 ts_tablet_manager.cc:1397] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20250109 20:53:12.273335 757 tablet_service.cc:1467] Processing CreateTablet for tablet f641db958b9e46ddbf9ed6d7f5ea37e9 (DEFAULT_TABLE table=default.Table_680621376a2449aeb32ebcd741a33b5b [id=b023caa314e647199dd4450ba76d7d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.273581 1158 raft_consensus.cc:357] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.273726 1158 raft_consensus.cc:383] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.273769 1158 raft_consensus.cc:738] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.273689 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f641db958b9e46ddbf9ed6d7f5ea37e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.273826 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 7889352df8064db7ad5180ac075397ff (DEFAULT_TABLE table=default.Table_85817f72c90941a2af0506a8759f34c3 [id=fd57b15ff31745ecafe1b05ab096c975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.273936 1158 consensus_queue.cc:260] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.274179 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7889352df8064db7ad5180ac075397ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.274320 1158 ts_tablet_manager.cc:1428] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:12.275682 1158 tablet_bootstrap.cc:492] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.277281 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9279bf08a26f4b3f8f55182fe7057cf4 (DEFAULT_TABLE table=default.Table_1db09bd8ba8c4c3fb3c525c18eb0e989 [id=e665e63d24984c89945406c1a5a56c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.277640 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9279bf08a26f4b3f8f55182fe7057cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.278715 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 37efddac3f8147e9a27ac0671f390d18 (DEFAULT_TABLE table=default.Table_e86358bf98744df4a2432305fcffde2b [id=e91423edbc354a93bafdfd83a4763f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.279182 1166 ts_tablet_manager.cc:1918] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.279300 1166 log.cc:1198] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/65570e42a9bd45bcba0a82abd12a9356
I20250109 20:53:12.279397 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37efddac3f8147e9a27ac0671f390d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.280242 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b7fee30e616b497f99d91ee199fa950a (DEFAULT_TABLE table=default.Table_0d846b274cdc42229ed64d25124b2560 [id=caefc0759f6042d8845c513375fcf11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.280586 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7fee30e616b497f99d91ee199fa950a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.281332 1166 ts_tablet_manager.cc:1939] T 65570e42a9bd45bcba0a82abd12a9356 P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:12.282282 1166 tablet_replica.cc:331] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:12.282392 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 65570e42a9bd45bcba0a82abd12a9356 (table default.Table_d773b9e15aff4f4fa339a5c6c2e70bd8 [id=426a2d55e76f4866b0f52ae174d1b643]) successfully deleted
I20250109 20:53:12.282622 1166 raft_consensus.cc:2238] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.282783 1166 raft_consensus.cc:2267] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.283340 1166 ts_tablet_manager.cc:1905] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.283735 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 2769e5906c604bd5b14df79db765ec61 (table default.Table_f59d59762b8b4f55b3123c4df5bfd12e [id=3c9451140e48432f8591841d4dfd8751]) successfully deleted
I20250109 20:53:12.283862 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 9279bf08a26f4b3f8f55182fe7057cf4 (DEFAULT_TABLE table=default.Table_1db09bd8ba8c4c3fb3c525c18eb0e989 [id=e665e63d24984c89945406c1a5a56c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.284134 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9279bf08a26f4b3f8f55182fe7057cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.284093 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 9d7587432d7345e4a225fc7d5c968c27 (table default.Table_b590793190454bb09e3e7dd36515f65a [id=4a57bf26eda04187bd4b854d402f3c19]) successfully deleted
I20250109 20:53:12.284265 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4870a667cd3e410590923c9e2497af54 (table default.Table_03ed60a072e941b6b5b9a8a477aa9020 [id=1e9c11e6b9294f56bb2efd10bef9af20]) successfully deleted
I20250109 20:53:12.284979 1020 tablet_service.cc:1467] Processing CreateTablet for tablet cb28bf037752421295e3e44785363882 (DEFAULT_TABLE table=default.Table_9d5da32f48064377ad7a7942e7122a8f [id=c0c7b6f20cd34261adf9e7dc9aaa0d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.285246 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb28bf037752421295e3e44785363882. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:12.285640 989 leader_election.cc:343] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.285887 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 6893f768a63b4abf8224621ccfb66a57 (DEFAULT_TABLE table=default.Table_507ede4faf0d4c83b5844d0247715180 [id=0a31e04681564cfb9224780341f0c47b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.285846 898 tablet_service.cc:1467] Processing CreateTablet for tablet 7889352df8064db7ad5180ac075397ff (DEFAULT_TABLE table=default.Table_85817f72c90941a2af0506a8759f34c3 [id=fd57b15ff31745ecafe1b05ab096c975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.286275 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7889352df8064db7ad5180ac075397ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.286554 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6893f768a63b4abf8224621ccfb66a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.288345 2627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8328960890b142248073cec450e957d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8328960890b142248073cec450e957d8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37941 } } peers { permanent_uuid: "0f066b28f8dd4e9cae5c64bd93feeb51" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 39231 } } peers { permanent_uuid: "36a29ab2002d4eb389cd11778988e1e9" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41583 } } }
I20250109 20:53:12.288527 2627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8 [sys.catalog]: This master's current role is: LEADER
I20250109 20:53:12.288887 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37efddac3f8147e9a27ac0671f390d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.289480 1158 tablet_bootstrap.cc:654] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.298596 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f641db958b9e46ddbf9ed6d7f5ea37e9 (DEFAULT_TABLE table=default.Table_680621376a2449aeb32ebcd741a33b5b [id=b023caa314e647199dd4450ba76d7d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.299162 1165 tablet_bootstrap.cc:492] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
W20250109 20:53:12.299273 990 leader_election.cc:343] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.299531 990 leader_election.cc:304] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.299678 1165 ts_tablet_manager.cc:1397] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.110s user 0.001s sys 0.006s
W20250109 20:53:12.299685 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 79492.8us
I20250109 20:53:12.299855 990 leader_election.cc:304] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:12.300084 2826 raft_consensus.cc:2743] T 2f53cef0adbb47d4840e4c11a6d0c8ac P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.300662 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6893f768a63b4abf8224621ccfb66a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.300542 1165 raft_consensus.cc:357] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.300992 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f641db958b9e46ddbf9ed6d7f5ea37e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.301050 1165 raft_consensus.cc:383] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.301117 1165 raft_consensus.cc:738] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.301362 1165 consensus_queue.cc:260] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.301909 1165 ts_tablet_manager.cc:1428] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.302134 1165 tablet_bootstrap.cc:492] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
W20250109 20:53:12.308558 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:42233, user_credentials={real_user=slave}} blocked reactor thread for 91260.1us
I20250109 20:53:12.311590 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eaa5bbfaf4c1c9d8cd1b1ead5f632" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:12.311769 1052 raft_consensus.cc:2463] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:12.312393 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f407b76780974cf7b3635fa1bf1585ec" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:12.312547 1052 raft_consensus.cc:2463] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:12.312965 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b285905342b3820f9549c8f0c4ad" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:12.313097 1052 raft_consensus.cc:2463] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:12.313685 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45cd5ffe7040e385d8cdfb97ed011a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:12.313813 1052 raft_consensus.cc:2463] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
W20250109 20:53:12.314414 989 leader_election.cc:343] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:12.314813 713 leader_election.cc:343] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 9170b3a6b9bb446b8a474ae4df249409
I20250109 20:53:12.314929 713 leader_election.cc:304] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.315121 2805 raft_consensus.cc:2743] T 9170b3a6b9bb446b8a474ae4df249409 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.315641 713 leader_election.cc:304] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.315837 2805 raft_consensus.cc:2798] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:12.315904 2805 raft_consensus.cc:491] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:12.315971 2805 raft_consensus.cc:3054] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.316187 713 leader_election.cc:304] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.316596 713 leader_election.cc:304] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.316989 2807 raft_consensus.cc:2798] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:12.317075 2807 raft_consensus.cc:491] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:12.317185 2807 raft_consensus.cc:3054] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.318457 2805 raft_consensus.cc:513] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.318995 2805 leader_election.cc:290] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.319181 2805 raft_consensus.cc:2798] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:12.319250 2805 raft_consensus.cc:491] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:12.319330 2805 raft_consensus.cc:3054] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.319624 2807 raft_consensus.cc:513] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.319677 888 tablet_service.cc:1467] Processing CreateTablet for tablet cb28bf037752421295e3e44785363882 (DEFAULT_TABLE table=default.Table_9d5da32f48064377ad7a7942e7122a8f [id=c0c7b6f20cd34261adf9e7dc9aaa0d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.320003 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb28bf037752421295e3e44785363882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.320168 2807 leader_election.cc:290] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.320703 713 leader_election.cc:343] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 374df8c6d1424697a3b3922f777e9870
I20250109 20:53:12.320895 713 leader_election.cc:304] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.321358 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f407b76780974cf7b3635fa1bf1585ec" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:12.321444 713 leader_election.cc:304] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.321645 2695 raft_consensus.cc:2743] T 374df8c6d1424697a3b3922f777e9870 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.321739 2807 raft_consensus.cc:2798] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:12.321830 2807 raft_consensus.cc:491] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:12.321905 2807 raft_consensus.cc:3054] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.322013 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eaa5bbfaf4c1c9d8cd1b1ead5f632" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:12.322415 712 leader_election.cc:343] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:12.322587 712 leader_election.cc:343] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.323669 2805 raft_consensus.cc:513] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.324175 2807 raft_consensus.cc:513] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.324589 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b285905342b3820f9549c8f0c4ad" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:12.325019 2807 leader_election.cc:290] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.325384 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45cd5ffe7040e385d8cdfb97ed011a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:12.325094 712 leader_election.cc:343] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:12.325853 712 leader_election.cc:343] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.328913 880 tablet_service.cc:1467] Processing CreateTablet for tablet 9279bf08a26f4b3f8f55182fe7057cf4 (DEFAULT_TABLE table=default.Table_1db09bd8ba8c4c3fb3c525c18eb0e989 [id=e665e63d24984c89945406c1a5a56c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.329259 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9279bf08a26f4b3f8f55182fe7057cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.329445 888 tablet_service.cc:1467] Processing CreateTablet for tablet f641db958b9e46ddbf9ed6d7f5ea37e9 (DEFAULT_TABLE table=default.Table_680621376a2449aeb32ebcd741a33b5b [id=b023caa314e647199dd4450ba76d7d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.329813 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f641db958b9e46ddbf9ed6d7f5ea37e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.331179 2818 raft_consensus.cc:491] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.331279 2818 raft_consensus.cc:513] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.331732 2818 leader_election.cc:290] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.333070 2805 leader_election.cc:290] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.334299 892 tablet_service.cc:1467] Processing CreateTablet for tablet 37efddac3f8147e9a27ac0671f390d18 (DEFAULT_TABLE table=default.Table_e86358bf98744df4a2432305fcffde2b [id=e91423edbc354a93bafdfd83a4763f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:12.334637 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37efddac3f8147e9a27ac0671f390d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:12.300232 2826 raft_consensus.cc:2798] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:12.334734 2826 raft_consensus.cc:491] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:12.334852 2826 raft_consensus.cc:3054] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.336901 1160 ts_tablet_manager.cc:1939] T fce062dc0ac144a19f8fa3eef4a099b4 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.341471 1052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63eaa5bbfaf4c1c9d8cd1b1ead5f632" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:12.341665 1052 raft_consensus.cc:3054] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.342228 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f407b76780974cf7b3635fa1bf1585ec" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:12.342432 1055 raft_consensus.cc:3054] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.342427 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a45cd5ffe7040e385d8cdfb97ed011a" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:12.342619 1053 raft_consensus.cc:3054] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.343060 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2420b285905342b3820f9549c8f0c4ad" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:12.343240 1051 raft_consensus.cc:3054] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.372195 1165 tablet_bootstrap.cc:654] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.373911 2818 raft_consensus.cc:491] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.374051 2818 raft_consensus.cc:513] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.374450 1160 tablet_replica.cc:331] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.374676 2818 leader_election.cc:290] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.374696 1160 raft_consensus.cc:2238] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.374984 1160 raft_consensus.cc:2267] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.375265 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet fce062dc0ac144a19f8fa3eef4a099b4 (table default.Table_986f5a94df1245f7930aca6efd8614a2 [id=2600eebbccdb4d098f1b48223a9edcd7]) successfully deleted
W20250109 20:53:12.375524 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 3e45125e7ca14225b67c0cf92f76b312 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e45125e7ca14225b67c0cf92f76b312 already in progress: creating tablet
I20250109 20:53:12.375656 1160 ts_tablet_manager.cc:1905] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:12.375694 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet a0aae7613aaf4248b4b8ada6e60fe8ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0aae7613aaf4248b4b8ada6e60fe8ea already in progress: creating tablet
W20250109 20:53:12.375837 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 58f3afa4595a41baa77723f14d6a4944 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58f3afa4595a41baa77723f14d6a4944 already in progress: creating tablet
W20250109 20:53:12.375983 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d758bb29a9294e9481bb15ca4b4429b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d758bb29a9294e9481bb15ca4b4429b0 already in progress: creating tablet
W20250109 20:53:12.376184 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet c50325c8a0e34acaa6e2912d3923d43f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c50325c8a0e34acaa6e2912d3923d43f already in progress: creating tablet
I20250109 20:53:12.377491 2826 raft_consensus.cc:513] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.378091 2826 leader_election.cc:290] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:12.381304 1052 raft_consensus.cc:2463] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:12.382445 1165 tablet_bootstrap.cc:492] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.382552 1165 ts_tablet_manager.cc:1397] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.080s user 0.000s sys 0.002s
I20250109 20:53:12.383186 1165 raft_consensus.cc:357] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.383344 1165 raft_consensus.cc:383] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.383383 1165 raft_consensus.cc:738] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.383533 1165 consensus_queue.cc:260] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.383849 1055 raft_consensus.cc:2463] T f407b76780974cf7b3635fa1bf1585ec P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:12.384003 1165 ts_tablet_manager.cc:1428] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250109 20:53:12.384224 1165 tablet_bootstrap.cc:492] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.384639 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c717e446858400789dd0e84b4fb60cd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:12.384855 778 raft_consensus.cc:3054] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.385774 1165 tablet_bootstrap.cc:654] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.387667 778 raft_consensus.cc:2463] T 8c717e446858400789dd0e84b4fb60cd P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:12.388089 990 leader_election.cc:304] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:12.389824 1160 ts_tablet_manager.cc:1918] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.389922 1160 log.cc:1198] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/4814361a1e854451a0ee7469cc6d447b
I20250109 20:53:12.390180 1160 ts_tablet_manager.cc:1939] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.390594 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 4814361a1e854451a0ee7469cc6d447b (table default.Table_ea3137b2ba3d44e3a715f18eb25dca01 [id=e4a51641b240493d9f69b9031ef5d4b1]) successfully deleted
I20250109 20:53:12.390654 1160 tablet_replica.cc:331] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.390813 1160 raft_consensus.cc:2238] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.390950 1160 raft_consensus.cc:2267] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.391577 1160 ts_tablet_manager.cc:1905] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.392845 1051 raft_consensus.cc:2463] T 2420b285905342b3820f9549c8f0c4ad P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
W20250109 20:53:12.395666 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ebaaefdc13e84f2e98f9926278adf637 (table default.Table_be8339044c5443cfb0ea29053fcbb2fd [id=45ba0ad8885f41b69c16d84c9399a720], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ebaaefdc13e84f2e98f9926278adf637 consensus state has no leader
I20250109 20:53:12.403062 1160 ts_tablet_manager.cc:1918] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.403162 1160 log.cc:1198] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/bd62a3ae64a143998bb1172b3a97c69f
I20250109 20:53:12.403403 1160 ts_tablet_manager.cc:1939] T bd62a3ae64a143998bb1172b3a97c69f P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.403738 1160 tablet_replica.cc:331] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.403915 1160 raft_consensus.cc:2238] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.404052 1160 raft_consensus.cc:2267] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.404774 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet bd62a3ae64a143998bb1172b3a97c69f (table default.Table_bbb3e86398ca4302b3f758d9a36837f9 [id=f6277ff896754951b965558aeeef0742]) successfully deleted
I20250109 20:53:12.405201 2822 raft_consensus.cc:2798] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:12.406478 1053 raft_consensus.cc:2463] T 8a45cd5ffe7040e385d8cdfb97ed011a P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:12.408061 1165 tablet_bootstrap.cc:492] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.408212 1165 ts_tablet_manager.cc:1397] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20250109 20:53:12.409215 1165 raft_consensus.cc:357] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.409464 1165 raft_consensus.cc:383] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.409530 1165 raft_consensus.cc:738] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.409804 1165 consensus_queue.cc:260] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.410426 1165 ts_tablet_manager.cc:1428] T 6569b450bb7a4a78922061d2ede981fe P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:12.410728 1165 tablet_bootstrap.cc:492] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.412353 1165 tablet_bootstrap.cc:654] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.413635 1160 ts_tablet_manager.cc:1905] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.414425 1176 tablet_bootstrap.cc:492] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:12.414598 1176 ts_tablet_manager.cc:1397] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.289s user 0.005s sys 0.000s
I20250109 20:53:12.415503 1176 raft_consensus.cc:357] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.415781 1176 raft_consensus.cc:383] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.415879 1176 raft_consensus.cc:738] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.416175 1176 consensus_queue.cc:260] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.419219 2913 tablet.cc:1721] T a85c36689d8f4b16b85db5ad3c84cac0 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.419373 1165 tablet_bootstrap.cc:492] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.419862 1165 ts_tablet_manager.cc:1397] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250109 20:53:12.420608 1165 raft_consensus.cc:357] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.420804 1165 raft_consensus.cc:383] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.420863 1165 raft_consensus.cc:738] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.421085 1165 consensus_queue.cc:260] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.421571 1165 ts_tablet_manager.cc:1428] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:12.421855 1165 tablet_bootstrap.cc:492] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.423175 1176 ts_tablet_manager.cc:1428] T e1faf5d79a9b43ffafc9c34a5937fcc0 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250109 20:53:12.423427 1176 tablet_bootstrap.cc:492] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:12.424608 2822 raft_consensus.cc:695] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:53:12.424687 2805 raft_consensus.cc:491] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.424928 2805 raft_consensus.cc:513] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.424949 2822 consensus_queue.cc:237] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.425143 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c717e446858400789dd0e84b4fb60cd" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:12.425915 2805 leader_election.cc:290] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.427199 1160 ts_tablet_manager.cc:1918] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.427321 1160 log.cc:1198] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/ea16fcc30f2c4b2eac53f20fcebd043f
I20250109 20:53:12.427569 1160 ts_tablet_manager.cc:1939] T ea16fcc30f2c4b2eac53f20fcebd043f P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.428015 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet ea16fcc30f2c4b2eac53f20fcebd043f (table default.Table_874fc86ef3cf4b27a752b1fa2e4c3b0d [id=35315fe696f042ee8700dd905bce5c5e]) successfully deleted
I20250109 20:53:12.428968 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2fcddba2ac4d82b2d25bfc64a5967e" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.429418 712 leader_election.cc:343] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.429960 1165 tablet_bootstrap.cc:654] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.431352 713 leader_election.cc:304] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.432085 713 leader_election.cc:304] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.432317 2692 raft_consensus.cc:2798] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:12.432422 2692 raft_consensus.cc:695] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:12.432670 2692 consensus_queue.cc:237] T d63eaa5bbfaf4c1c9d8cd1b1ead5f632 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.432860 713 leader_election.cc:304] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.433109 2807 raft_consensus.cc:2798] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:12.433697 2754 raft_consensus.cc:2798] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:12.433840 2754 raft_consensus.cc:695] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:12.434051 2754 consensus_queue.cc:237] T 2420b285905342b3820f9549c8f0c4ad P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.437007 713 leader_election.cc:304] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.444983 2912 raft_consensus.cc:491] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.445430 2912 raft_consensus.cc:513] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.447125 1165 tablet_bootstrap.cc:492] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.447300 1165 ts_tablet_manager.cc:1397] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250109 20:53:12.447854 2917 raft_consensus.cc:2798] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:12.447973 2917 raft_consensus.cc:695] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:12.448069 1165 raft_consensus.cc:357] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.448235 1165 raft_consensus.cc:383] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.448282 1165 raft_consensus.cc:738] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.448232 2917 consensus_queue.cc:237] T 8a45cd5ffe7040e385d8cdfb97ed011a P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.448694 1160 tablet_replica.cc:331] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.448927 1160 raft_consensus.cc:2238] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.449069 1160 raft_consensus.cc:2267] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.449206 2807 raft_consensus.cc:695] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:12.449774 2807 consensus_queue.cc:237] T f407b76780974cf7b3635fa1bf1585ec P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.450860 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beee9825fa0240e88029bf32ace0e187" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:12.451336 990 leader_election.cc:343] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.453531 1176 tablet_bootstrap.cc:654] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.453886 1160 ts_tablet_manager.cc:1905] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.457607 1165 consensus_queue.cc:260] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.457913 2912 leader_election.cc:290] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:12.458086 1165 ts_tablet_manager.cc:1428] T 404ee892ab79427baaef6d19c57699d0 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250109 20:53:12.458346 1165 tablet_bootstrap.cc:492] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.459184 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beee9825fa0240e88029bf32ace0e187" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.460083 1165 tablet_bootstrap.cc:654] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.460950 2805 raft_consensus.cc:491] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.461097 2805 raft_consensus.cc:513] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.461635 1158 tablet_bootstrap.cc:492] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.461840 1158 ts_tablet_manager.cc:1397] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.186s user 0.020s sys 0.044s
I20250109 20:53:12.461937 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db4e85cd4094cf0b5b699402346b980" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.462003 2805 leader_election.cc:290] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.462416 712 leader_election.cc:343] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.462692 1158 raft_consensus.cc:357] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.462996 1158 raft_consensus.cc:383] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.463054 1158 raft_consensus.cc:738] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.463238 1158 consensus_queue.cc:260] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.463249 2826 consensus_queue.cc:1035] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:12.463689 1158 ts_tablet_manager.cc:1428] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:12.463963 1158 tablet_bootstrap.cc:492] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.465664 1158 tablet_bootstrap.cc:654] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.471796 1160 ts_tablet_manager.cc:1918] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.471904 1160 log.cc:1198] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/de8ea5ea7fa44866b6d7160466f5a896
I20250109 20:53:12.472186 1160 ts_tablet_manager.cc:1939] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
W20250109 20:53:12.478148 989 leader_election.cc:343] T 8c717e446858400789dd0e84b4fb60cd P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:12.479336 989 leader_election.cc:343] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.479713 989 leader_election.cc:304] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.482407 2822 raft_consensus.cc:2743] T beee9825fa0240e88029bf32ace0e187 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.482867 1158 tablet_bootstrap.cc:492] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.483021 1158 ts_tablet_manager.cc:1397] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250109 20:53:12.483042 2911 tablet.cc:1721] T a85c36689d8f4b16b85db5ad3c84cac0 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.484097 1158 raft_consensus.cc:357] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.484287 1158 raft_consensus.cc:383] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.484377 1158 raft_consensus.cc:738] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.484557 1158 consensus_queue.cc:260] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.485102 1158 ts_tablet_manager.cc:1428] T e457baa02f4e48f8a627ffdb05267fc8 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.485385 1158 tablet_bootstrap.cc:492] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.486881 1158 tablet_bootstrap.cc:654] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.488299 1165 tablet_bootstrap.cc:492] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.488462 1165 ts_tablet_manager.cc:1397] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.005s
I20250109 20:53:12.489467 1165 raft_consensus.cc:357] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.489706 1165 raft_consensus.cc:383] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.489758 1165 raft_consensus.cc:738] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.490049 1165 consensus_queue.cc:260] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.490592 1165 ts_tablet_manager.cc:1428] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:12.491107 1166 ts_tablet_manager.cc:1918] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.491225 1166 log.cc:1198] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/ade6b4ea4ca44f3eb88d3911b0bbb70b
I20250109 20:53:12.491443 1160 tablet_replica.cc:331] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.491547 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet de8ea5ea7fa44866b6d7160466f5a896 (table default.Table_64ebceb89d2c4e8291bc096091578453 [id=94d89d19277c4e3fbf4ba420ebe0965c]) successfully deleted
I20250109 20:53:12.491631 1160 raft_consensus.cc:2238] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:12.491593 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36949, user_credentials={real_user=slave}} blocked reactor thread for 54533.6us
I20250109 20:53:12.491755 1160 raft_consensus.cc:2267] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:12.492241 713 leader_election.cc:343] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 5b2fcddba2ac4d82b2d25bfc64a5967e
I20250109 20:53:12.492369 713 leader_election.cc:304] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.492553 1160 ts_tablet_manager.cc:1905] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.492910 2920 raft_consensus.cc:2743] T 5b2fcddba2ac4d82b2d25bfc64a5967e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.495847 1165 tablet_bootstrap.cc:492] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.497409 1165 tablet_bootstrap.cc:654] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.506456 1166 ts_tablet_manager.cc:1939] T ade6b4ea4ca44f3eb88d3911b0bbb70b P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
I20250109 20:53:12.507535 1158 tablet_bootstrap.cc:492] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.507666 1158 ts_tablet_manager.cc:1397] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20250109 20:53:12.508363 1158 raft_consensus.cc:357] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.508548 1158 raft_consensus.cc:383] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.508651 1158 raft_consensus.cc:738] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.508957 1158 consensus_queue.cc:260] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.509475 1158 ts_tablet_manager.cc:1428] T a1ee36f3be324b14868976298ca26118 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250109 20:53:12.509716 1158 tablet_bootstrap.cc:492] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.510998 1158 tablet_bootstrap.cc:654] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.512583 1160 ts_tablet_manager.cc:1918] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.512683 1160 log.cc:1198] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/0019a01afbcc452a9a9374c92dc131f3
I20250109 20:53:12.513813 1160 ts_tablet_manager.cc:1939] T 0019a01afbcc452a9a9374c92dc131f3 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.514379 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 0019a01afbcc452a9a9374c92dc131f3 (table default.Table_6d708581ee5c4179af45f5e8ce12bec5 [id=7a4b26558b3b43d097a1c01b0504b037]) successfully deleted
I20250109 20:53:12.522146 1158 tablet_bootstrap.cc:492] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.522392 1158 ts_tablet_manager.cc:1397] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20250109 20:53:12.523211 1158 raft_consensus.cc:357] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.523553 1158 raft_consensus.cc:383] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.523663 1158 raft_consensus.cc:738] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.523938 1158 consensus_queue.cc:260] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.493785 1176 tablet_bootstrap.cc:492] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:12.524623 1176 ts_tablet_manager.cc:1397] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.000s
I20250109 20:53:12.525247 1158 ts_tablet_manager.cc:1428] T 75cdf5b737504d4e8f598cd2b265192e P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:53:12.525663 1176 raft_consensus.cc:357] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.525884 1176 raft_consensus.cc:383] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.525941 1176 raft_consensus.cc:738] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.526238 1158 tablet_bootstrap.cc:492] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.526134 1176 consensus_queue.cc:260] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.526750 1176 ts_tablet_manager.cc:1428] T 274a58191d00475e8da9ff72dd33524f P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:12.527021 1176 tablet_bootstrap.cc:492] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:12.527846 1158 tablet_bootstrap.cc:654] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.528533 1176 tablet_bootstrap.cc:654] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.530247 712 consensus_peers.cc:487] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c5e9b31b27949c2948cb64b6124ee72. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:12.534149 1158 tablet_bootstrap.cc:492] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.534365 1158 ts_tablet_manager.cc:1397] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20250109 20:53:12.535182 1158 raft_consensus.cc:357] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.535466 1158 raft_consensus.cc:383] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.535566 1158 raft_consensus.cc:738] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.535836 1158 consensus_queue.cc:260] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.536552 1158 ts_tablet_manager.cc:1428] T c70ddcc69b2b42fcbe133978dd3ac05b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.536909 1158 tablet_bootstrap.cc:492] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.538391 1158 tablet_bootstrap.cc:654] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.546343 1158 tablet_bootstrap.cc:492] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.546526 1158 ts_tablet_manager.cc:1397] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20250109 20:53:12.547981 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet d63eaa5bbfaf4c1c9d8cd1b1ead5f632 (table default.Table_e05a32da191c46ffb04a2fa03c7920f5 [id=06fb75d530e343559e65bc14300c15b9]). Expected version 1 got 0
W20250109 20:53:12.548380 713 leader_election.cc:343] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 2db4e85cd4094cf0b5b699402346b980
I20250109 20:53:12.548481 713 leader_election.cc:304] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.548730 2807 raft_consensus.cc:2743] T 2db4e85cd4094cf0b5b699402346b980 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.548986 2807 consensus_queue.cc:1035] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:12.550191 1166 tablet_replica.cc:331] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: stopping tablet replica
I20250109 20:53:12.550406 1166 raft_consensus.cc:2238] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.550591 1166 raft_consensus.cc:2267] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.551097 1166 ts_tablet_manager.cc:1905] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.553674 1158 raft_consensus.cc:357] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.553833 1158 raft_consensus.cc:383] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.548914 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 156fbcdb4c9b4f83a96dad732a9a0964
I20250109 20:53:12.553874 1158 raft_consensus.cc:738] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.554033 1158 consensus_queue.cc:260] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.554447 1158 ts_tablet_manager.cc:1428] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250109 20:53:12.554400 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7d5333c1118343aeb596d21852bbedbf" } new_table_name: "default.Table_84e40522a2db4734a4c94af5979eb88e" modify_external_catalogs: true
I20250109 20:53:12.554662 1158 tablet_bootstrap.cc:492] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.549088 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet d11e17f3391740b79ffa9d8a8c9bf400
I20250109 20:53:12.555908 1158 tablet_bootstrap.cc:654] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.555966 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 156fbcdb4c9b4f83a96dad732a9a0964 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.556285 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_168790ba2811484e977ede30bf28842b" } new_table_name: "default.Table_30556d9456d24757849a23be2fe298b6" modify_external_catalogs: true
W20250109 20:53:12.557194 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 156fbcdb4c9b4f83a96dad732a9a0964 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 156fbcdb4c9b4f83a96dad732a9a0964 already in progress: creating tablet
I20250109 20:53:12.557343 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d11e17f3391740b79ffa9d8a8c9bf400 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:12.557889 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet d11e17f3391740b79ffa9d8a8c9bf400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d11e17f3391740b79ffa9d8a8c9bf400 already in progress: creating tablet
I20250109 20:53:12.549437 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cfdca0092aa74d3e805ce6c6132ca08f" } modify_external_catalogs: true
I20250109 20:53:12.558547 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_cfdca0092aa74d3e805ce6c6132ca08f" } modify_external_catalogs: true
I20250109 20:53:12.550169 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet ade6b4ea4ca44f3eb88d3911b0bbb70b (table default.Table_e0cf5166142546989eb825944cae0691 [id=ebdd987074c643c6a5de34374b5002d4]) successfully deleted
W20250109 20:53:12.550400 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ad5f7eaea8c46b6ba0f83753112094e consensus state has no leader
I20250109 20:53:12.560461 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_fc10e7e4961046c1b63ae98f857f8ff1" } new_table_name: "default.Table_21d80500f6f64d1db253deb08160d47f" modify_external_catalogs: true
W20250109 20:53:12.561089 989 consensus_peers.cc:487] T c0860554cac64b8882e39fa62462b1aa P 5fcc647ed479491b88e5ffacf2927a1a -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:12.550699 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7768e94e2bff455eb490b7cd4e6f1fbb" } modify_external_catalogs: true
I20250109 20:53:12.561730 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7768e94e2bff455eb490b7cd4e6f1fbb" } modify_external_catalogs: true
I20250109 20:53:12.562758 452 catalog_manager.cc:5618] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465) does not have the latest schema for tablet 50039da210df47ab8476e28387c8a802 (table default.Table_422152ed650445aeb8f7af679f9c1d12 [id=b296e8173f88419a8a14ac395611b4bc]). Expected version 3 got 0
I20250109 20:53:12.553287 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a9d811c825cf4a0c9ec6a3587a72e497"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.563961 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9d811c825cf4a0c9ec6a3587a72e497 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:12.564594 712 consensus_peers.cc:487] T f7b4c038573a4afa879ecbcba75c35a7 P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:12.553831 453 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet f5371e9438ed42c380640d0608ee2578
I20250109 20:53:12.565514 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_64fcbf9328cd46808d4c690bbb864ee4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.565951 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64fcbf9328cd46808d4c690bbb864ee4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.563169 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6e2a1fc9aaf84f169ee8e33b0bf2315e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.567960 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e2a1fc9aaf84f169ee8e33b0bf2315e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.551401 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_848a8bb910414b7faca9f9f52cbfc502" } new_table_name: "default.Table_eabeade2ce7f418cbcaaf2a9f343ae4d" modify_external_catalogs: true
I20250109 20:53:12.569725 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d11e17f3391740b79ffa9d8a8c9bf400 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.572245 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_091032cc54c64921bd5d73ac649efb54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.572664 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_091032cc54c64921bd5d73ac649efb54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.573045 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d11e17f3391740b79ffa9d8a8c9bf400 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.573267 1158 tablet_bootstrap.cc:492] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.573316 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 156fbcdb4c9b4f83a96dad732a9a0964 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.573385 1158 ts_tablet_manager.cc:1397] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250109 20:53:12.573702 757 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c88f42c15d94099903178218bfac1e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:59 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.573908 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db4e85cd4094cf0b5b699402346b980 with delete_type TABLET_DATA_DELETED (replaced by tablet b54b66ca10114330b3485f67eb076064) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.574028 1158 raft_consensus.cc:357] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.574199 1158 raft_consensus.cc:383] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.574242 1158 raft_consensus.cc:738] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.574420 1158 consensus_queue.cc:260] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.574831 1158 ts_tablet_manager.cc:1428] T beca5f1ed9d3434fb7de90cd74a6f09b P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:12.575066 1158 tablet_bootstrap.cc:492] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.575193 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 156fbcdb4c9b4f83a96dad732a9a0964 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.576510 1158 tablet_bootstrap.cc:654] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.577164 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 374df8c6d1424697a3b3922f777e9870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:52:54 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.583850 1158 tablet_bootstrap.cc:492] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.583976 1158 ts_tablet_manager.cc:1397] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250109 20:53:12.584535 1158 raft_consensus.cc:357] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.584686 1158 raft_consensus.cc:383] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.584748 1158 raft_consensus.cc:738] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.584936 1158 consensus_queue.cc:260] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.585454 1158 ts_tablet_manager.cc:1428] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250109 20:53:12.585668 1158 tablet_bootstrap.cc:492] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.587184 1158 tablet_bootstrap.cc:654] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.600553 854 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:35465, user_credentials={real_user=slave}} blocked reactor thread for 350722us
I20250109 20:53:12.613634 1158 tablet_bootstrap.cc:492] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.613777 1158 ts_tablet_manager.cc:1397] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.001s
I20250109 20:53:12.614441 1158 raft_consensus.cc:357] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.614619 1158 raft_consensus.cc:383] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.614658 1158 raft_consensus.cc:738] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.614845 1158 consensus_queue.cc:260] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.615429 1158 ts_tablet_manager.cc:1428] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.616010 1158 tablet_bootstrap.cc:492] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.617426 1158 tablet_bootstrap.cc:654] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.627909 1166 ts_tablet_manager.cc:1918] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.632295 2807 raft_consensus.cc:491] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.632437 2807 raft_consensus.cc:513] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.633025 2807 leader_election.cc:290] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.634028 713 leader_election.cc:343] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: f2383d9770834b1aae290308b4185299
I20250109 20:53:12.608493 1165 tablet_bootstrap.cc:492] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.636633 1165 ts_tablet_manager.cc:1397] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.000s
I20250109 20:53:12.637583 1165 raft_consensus.cc:357] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.637763 1165 raft_consensus.cc:383] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.637809 1165 raft_consensus.cc:738] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.637988 1165 consensus_queue.cc:260] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.638489 1165 ts_tablet_manager.cc:1428] T face19003d0042669e3f22553f9fe19a P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:12.638746 1165 tablet_bootstrap.cc:492] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.633577 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2383d9770834b1aae290308b4185299" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.641676 712 leader_election.cc:343] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.641806 712 leader_election.cc:304] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.642190 1165 tablet_bootstrap.cc:654] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.642284 2920 raft_consensus.cc:2743] T f2383d9770834b1aae290308b4185299 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.645287 1158 tablet_bootstrap.cc:492] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.645409 1158 ts_tablet_manager.cc:1397] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20250109 20:53:12.646024 1158 raft_consensus.cc:357] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.646241 1158 raft_consensus.cc:383] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.646288 1158 raft_consensus.cc:738] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.646448 1158 consensus_queue.cc:260] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.648978 1158 ts_tablet_manager.cc:1428] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:53:12.650064 1158 tablet_bootstrap.cc:492] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
W20250109 20:53:12.650703 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d0059863d73460a9e7d12c46a7616f3 consensus state has no leader
I20250109 20:53:12.652333 1158 tablet_bootstrap.cc:654] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.652921 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 7558fa96ea4e4fdd81b783880649d5a6
I20250109 20:53:12.655190 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 7558fa96ea4e4fdd81b783880649d5a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.655581 1165 tablet_bootstrap.cc:492] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.655090 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7558fa96ea4e4fdd81b783880649d5a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.655723 1165 ts_tablet_manager.cc:1397] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20250109 20:53:12.654887 1166 log.cc:1198] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-2/wal/wals/2f5609eaf86d4e8ea3d3da41243c927f
I20250109 20:53:12.656111 1166 ts_tablet_manager.cc:1939] T 2f5609eaf86d4e8ea3d3da41243c927f P 5fcc647ed479491b88e5ffacf2927a1a: Deleting consensus metadata
W20250109 20:53:12.656488 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 413341019ff94a10896d956f5a03c8ae consensus state has no leader
I20250109 20:53:12.656769 450 master_service.cc:946] ReplaceTablet: received request to replace tablet edc803f2936b4538a5e70ef34e5cc5d4 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:12.656790 1165 raft_consensus.cc:357] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.657018 1165 raft_consensus.cc:383] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.657076 1165 raft_consensus.cc:738] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.657353 1165 consensus_queue.cc:260] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.636605 2934 tablet.cc:1721] T a85c36689d8f4b16b85db5ad3c84cac0 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.657853 1165 ts_tablet_manager.cc:1428] T bdcb6780422541c89343e98e02938d0e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:12.658118 1165 tablet_bootstrap.cc:492] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.659024 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 33ffcbca06534d03ab82c02e16608d66
I20250109 20:53:12.659533 451 master_service.cc:946] ReplaceTablet: received request to replace tablet e54e9be016584fde911c4524969d0789 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:12.659878 1165 tablet_bootstrap.cc:654] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.660701 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ffcbca06534d03ab82c02e16608d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.661635 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ffcbca06534d03ab82c02e16608d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.662918 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 946c48edb09e43e2bac78a1540946d92 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:12.639091 1176 tablet_bootstrap.cc:492] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:12.663286 1176 ts_tablet_manager.cc:1397] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.136s user 0.006s sys 0.000s
I20250109 20:53:12.663981 1176 raft_consensus.cc:357] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.664184 1176 raft_consensus.cc:383] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.664248 1176 raft_consensus.cc:738] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.664440 1176 consensus_queue.cc:260] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.665241 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 7558fa96ea4e4fdd81b783880649d5a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.665445 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ffcbca06534d03ab82c02e16608d66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:12 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.665926 1176 ts_tablet_manager.cc:1428] T 2e1f70532d3045499f2c3ef554740247 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250109 20:53:12.666236 1176 tablet_bootstrap.cc:492] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:12.667379 1165 tablet_bootstrap.cc:492] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.667518 1165 ts_tablet_manager.cc:1397] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250109 20:53:12.667954 1176 tablet_bootstrap.cc:654] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.668015 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 7558fa96ea4e4fdd81b783880649d5a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7558fa96ea4e4fdd81b783880649d5a6 already in progress: creating tablet
W20250109 20:53:12.668252 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 33ffcbca06534d03ab82c02e16608d66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33ffcbca06534d03ab82c02e16608d66 already in progress: creating tablet
I20250109 20:53:12.668251 1165 raft_consensus.cc:357] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.668412 1165 raft_consensus.cc:383] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.668471 1165 raft_consensus.cc:738] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.668669 1165 consensus_queue.cc:260] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.669638 1165 ts_tablet_manager.cc:1428] T cc54e97e91c54afea098624bd85fcc6e P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250109 20:53:12.669981 1165 tablet_bootstrap.cc:492] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.671821 1165 tablet_bootstrap.cc:654] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.676591 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3911b54a040144baadfc4e0088d58e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-09 20:53:09 UTC) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.676756 441 catalog_manager.cc:4872] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): tablet 2f5609eaf86d4e8ea3d3da41243c927f (table default.Table_141b5c9b04d6457f8ebe492f7a7fb482 [id=2e25adc0595647c88f6cf27458078a59]) successfully deleted
I20250109 20:53:12.676829 1160 tablet_replica.cc:331] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: stopping tablet replica
I20250109 20:53:12.677002 1160 raft_consensus.cc:2238] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.677131 1160 raft_consensus.cc:2267] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.677623 1160 ts_tablet_manager.cc:1905] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250109 20:53:12.677708 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet d11e17f3391740b79ffa9d8a8c9bf400 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d11e17f3391740b79ffa9d8a8c9bf400 already in progress: creating tablet
W20250109 20:53:12.678438 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 156fbcdb4c9b4f83a96dad732a9a0964 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 156fbcdb4c9b4f83a96dad732a9a0964 already in progress: creating tablet
W20250109 20:53:12.678782 854 leader_election.cc:343] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: de8ea5ea7fa44866b6d7160466f5a896
I20250109 20:53:12.678892 854 leader_election.cc:304] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:12.678937 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 7558fa96ea4e4fdd81b783880649d5a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7558fa96ea4e4fdd81b783880649d5a6 already in progress: creating tablet
W20250109 20:53:12.679044 854 raft_consensus.cc:2681] T de8ea5ea7fa44866b6d7160466f5a896 P 44d473955919435cb679edd5f8afe2c2: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250109 20:53:12.679312 854 leader_election.cc:343] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: c865faed94c3493ea638826bdcd4253d
W20250109 20:53:12.679458 854 leader_election.cc:343] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 4814361a1e854451a0ee7469cc6d447b
W20250109 20:53:12.679454 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 33ffcbca06534d03ab82c02e16608d66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33ffcbca06534d03ab82c02e16608d66 already in progress: creating tablet
I20250109 20:53:12.680850 2826 raft_consensus.cc:491] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.681025 2826 raft_consensus.cc:513] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.681674 2826 leader_election.cc:290] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:12.683283 1160 ts_tablet_manager.cc:1918] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.683230 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44af81df98ae4b05927dab7b4246de9f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:12.683362 1160 log.cc:1198] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-1/wal/wals/3911b54a040144baadfc4e0088d58e64
I20250109 20:53:12.683418 778 raft_consensus.cc:2463] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 0.
I20250109 20:53:12.683621 1160 ts_tablet_manager.cc:1939] T 3911b54a040144baadfc4e0088d58e64 P 44d473955919435cb679edd5f8afe2c2: Deleting consensus metadata
I20250109 20:53:12.683789 990 leader_election.cc:304] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:12.684082 2826 raft_consensus.cc:2798] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:12.684156 2826 raft_consensus.cc:491] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:12.684216 2826 raft_consensus.cc:3054] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.684194 440 catalog_manager.cc:4872] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): tablet 3911b54a040144baadfc4e0088d58e64 (table default.Table_1eee2fc76a024644bdd78f7fd1ef0e62 [id=24b47aa4d1f44087a21a07eba9320ab1]) successfully deleted
I20250109 20:53:12.686579 2826 raft_consensus.cc:513] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.687692 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44af81df98ae4b05927dab7b4246de9f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f"
I20250109 20:53:12.687884 778 raft_consensus.cc:3054] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
W20250109 20:53:12.688290 854 leader_election.cc:343] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: c865faed94c3493ea638826bdcd4253d
I20250109 20:53:12.688378 854 leader_election.cc:304] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.690348 2818 raft_consensus.cc:2743] T c865faed94c3493ea638826bdcd4253d P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.690431 778 raft_consensus.cc:2463] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fcc647ed479491b88e5ffacf2927a1a in term 1.
I20250109 20:53:12.690986 990 leader_election.cc:304] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters:
I20250109 20:53:12.691202 2912 raft_consensus.cc:2798] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:12.691303 2912 raft_consensus.cc:695] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [term 1 LEADER]: Becoming Leader. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Running, Role: LEADER
I20250109 20:53:12.691501 2912 consensus_queue.cc:237] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.691879 2826 leader_election.cc:290] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465)
I20250109 20:53:12.694147 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44af81df98ae4b05927dab7b4246de9f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
I20250109 20:53:12.694494 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44af81df98ae4b05927dab7b4246de9f" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.694653 854 leader_election.cc:343] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 4814361a1e854451a0ee7469cc6d447b
I20250109 20:53:12.694748 854 leader_election.cc:304] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:12.694897 854 raft_consensus.cc:2681] T 4814361a1e854451a0ee7469cc6d447b P 44d473955919435cb679edd5f8afe2c2: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250109 20:53:12.695089 989 leader_election.cc:343] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
W20250109 20:53:12.695380 989 leader_election.cc:343] T 44af81df98ae4b05927dab7b4246de9f P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.700181 1165 tablet_bootstrap.cc:492] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.700330 1165 ts_tablet_manager.cc:1397] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20250109 20:53:12.701131 2807 raft_consensus.cc:491] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.701159 1165 raft_consensus.cc:357] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.701311 1165 raft_consensus.cc:383] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.701356 1165 raft_consensus.cc:738] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.701431 2807 raft_consensus.cc:513] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.701511 1165 consensus_queue.cc:260] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.701941 1165 ts_tablet_manager.cc:1428] T f7b4c038573a4afa879ecbcba75c35a7 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250109 20:53:12.702189 1165 tablet_bootstrap.cc:492] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.702538 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d13168ca13c400a8a2d9608293ff8c3" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.702981 712 leader_election.cc:343] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.703318 2807 leader_election.cc:290] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.703748 1165 tablet_bootstrap.cc:654] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.703953 713 leader_election.cc:343] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 9d13168ca13c400a8a2d9608293ff8c3
I20250109 20:53:12.704062 713 leader_election.cc:304] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.704272 2807 raft_consensus.cc:2743] T 9d13168ca13c400a8a2d9608293ff8c3 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.708453 2913 tablet.cc:1721] T 2f494150713b4dde9399d53d57567983 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.708863 2911 tablet.cc:1721] T 2f494150713b4dde9399d53d57567983 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.710354 2934 tablet.cc:1721] T 2f494150713b4dde9399d53d57567983 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.719010 1157 ts_tablet_manager.cc:1905] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.724248 1157 ts_tablet_manager.cc:1918] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.726380 1165 tablet_bootstrap.cc:492] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.726553 1165 ts_tablet_manager.cc:1397] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20250109 20:53:12.727557 1165 raft_consensus.cc:357] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.727797 1165 raft_consensus.cc:383] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.727862 1165 raft_consensus.cc:738] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.728092 1165 consensus_queue.cc:260] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.728235 454 catalog_manager.cc:5618] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233) does not have the latest schema for tablet cc54e97e91c54afea098624bd85fcc6e (table default.Table_bc4981434b6743fc93063e84ec2ed645 [id=28723b313bd34951826646276c761571]). Expected version 3 got 0
I20250109 20:53:12.728673 1165 ts_tablet_manager.cc:1428] T deedac68322e460b8d62adbe849de3f2 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.729038 1165 tablet_bootstrap.cc:492] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.731086 1165 tablet_bootstrap.cc:654] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.734323 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c16437bb00e410a9fcb774aa3d32172 consensus state has no leader
I20250109 20:53:12.735615 455 master_service.cc:946] ReplaceTablet: received request to replace tablet f948f9c3f77d498a9043382d6d3ba72e from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:12.737215 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet 946c48edb09e43e2bac78a1540946d92
I20250109 20:53:12.737673 447 catalog_manager.cc:6596] ReplaceTablet: tablet 946c48edb09e43e2bac78a1540946d92 deleted and replaced by tablet 1dc55876ecc84a29a0be3bd62b6242df
I20250109 20:53:12.738197 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 946c48edb09e43e2bac78a1540946d92 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dc55876ecc84a29a0be3bd62b6242df) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.738544 1165 tablet_bootstrap.cc:492] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.738674 1165 ts_tablet_manager.cc:1397] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20250109 20:53:12.739452 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 946c48edb09e43e2bac78a1540946d92 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dc55876ecc84a29a0be3bd62b6242df) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.739468 1165 raft_consensus.cc:357] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.739674 1165 raft_consensus.cc:383] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.739724 1165 raft_consensus.cc:738] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.739914 1165 consensus_queue.cc:260] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
W20250109 20:53:12.740006 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet 946c48edb09e43e2bac78a1540946d92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 946c48edb09e43e2bac78a1540946d92 already in progress: creating tablet
W20250109 20:53:12.740069 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet 946c48edb09e43e2bac78a1540946d92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 946c48edb09e43e2bac78a1540946d92 already in progress: creating tablet
I20250109 20:53:12.740466 1165 ts_tablet_manager.cc:1428] T 0a2efcc2a6ec4b109845d8087491ca9f P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:12.741179 1165 tablet_bootstrap.cc:492] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.742904 1165 tablet_bootstrap.cc:654] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.750183 1165 tablet_bootstrap.cc:492] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.750298 1165 ts_tablet_manager.cc:1397] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20250109 20:53:12.750401 1157 log.cc:1198] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/527776a5d742446eb8fe61c3c855021f
W20250109 20:53:12.724946 830 tablet.cc:2367] T 44af81df98ae4b05927dab7b4246de9f P 09833ea46b7141d58c3899dace69e49f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:53:12.753862 1157 ts_tablet_manager.cc:1939] T 527776a5d742446eb8fe61c3c855021f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:12.754711 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1682) took 1230 ms. Trace:
I20250109 20:53:12.754778 1157 rpcz_store.cc:276] 0109 20:53:11.524727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:11.524812 (+ 85us) service_pool.cc:225] Handling call
0109 20:53:12.152878 (+628066us) ts_tablet_manager.cc:1145] Deleting tablet 527776a5d742446eb8fe61c3c855021f
0109 20:53:12.152887 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:12.721788 (+568901us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:12.724179 (+ 2391us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:12.754694 (+ 30515us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3816,"tablet-delete.queue_time_us":620885}
I20250109 20:53:12.755338 1157 tablet_replica.cc:331] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:12.755429 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 527776a5d742446eb8fe61c3c855021f (table default.Table_748cfdfa67a04dc8bf878fccb596b642 [id=c82368e2d31f4654803844337d1d41a8]) successfully deleted
I20250109 20:53:12.755970 1165 raft_consensus.cc:357] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.756161 1165 raft_consensus.cc:383] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.756209 1165 raft_consensus.cc:738] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.756392 1165 consensus_queue.cc:260] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.756858 1165 ts_tablet_manager.cc:1428] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250109 20:53:12.757859 2911 tablet.cc:1721] T d3681efa8785480ca65a51b4b40448f1 P 09833ea46b7141d58c3899dace69e49f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.758220 1165 tablet_bootstrap.cc:492] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.758466 2913 tablet.cc:1721] T d3681efa8785480ca65a51b4b40448f1 P 44d473955919435cb679edd5f8afe2c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.758917 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_916c7dba3cba475c92b35be9fd65843d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:12.759330 2958 raft_consensus.cc:491] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250109 20:53:12.759454 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_916c7dba3cba475c92b35be9fd65843d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.759503 2958 raft_consensus.cc:513] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.759938 1165 tablet_bootstrap.cc:654] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.760107 2958 leader_election.cc:290] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.761075 854 leader_election.cc:343] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 72a6cf525cb44c428f7a39a25c4e62dc
I20250109 20:53:12.763849 1165 tablet_bootstrap.cc:492] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.763949 1165 ts_tablet_manager.cc:1397] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250109 20:53:12.764549 1165 raft_consensus.cc:357] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.764688 1165 raft_consensus.cc:383] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.764746 1165 raft_consensus.cc:738] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.764894 1165 consensus_queue.cc:260] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.765290 1165 ts_tablet_manager.cc:1428] T 6461f153a5994412ba05c3b687963fd1 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250109 20:53:12.765484 1165 tablet_bootstrap.cc:492] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.766570 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_0d044688ecbf4a0cad83b24d4b3168ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:12.767120 1165 tablet_bootstrap.cc:654] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.767341 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d044688ecbf4a0cad83b24d4b3168ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.729813 2805 raft_consensus.cc:491] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.770098 2805 raft_consensus.cc:513] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.770570 2934 tablet.cc:1721] T d3681efa8785480ca65a51b4b40448f1 P 5fcc647ed479491b88e5ffacf2927a1a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250109 20:53:12.770741 2805 leader_election.cc:290] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.771135 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_c8b1137d9a404f29ab8cce24e0c18d8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.771641 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8b1137d9a404f29ab8cce24e0c18d8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:12.771620 713 leader_election.cc:343] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: b56d2c0706c84195acceb97ef4985c14
I20250109 20:53:12.771697 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_fa90ceb799cb4255b3cc196026031364"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.772078 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa90ceb799cb4255b3cc196026031364 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.772365 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_1dc88f92e6d14ca4b35171ed46b785ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.772773 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dc88f92e6d14ca4b35171ed46b785ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.741917 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 946c48edb09e43e2bac78a1540946d92 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dc55876ecc84a29a0be3bd62b6242df) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.747221 2807 raft_consensus.cc:491] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.774637 2807 raft_consensus.cc:513] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.775190 2807 leader_election.cc:290] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.775835 713 leader_election.cc:343] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 31de9638669040db9674d3aa7df32103
I20250109 20:53:12.776541 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b56d2c0706c84195acceb97ef4985c14" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.776921 1157 raft_consensus.cc:2238] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250109 20:53:12.777091 1157 raft_consensus.cc:2267] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250109 20:53:12.777415 712 leader_election.cc:343] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.777524 712 leader_election.cc:304] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.777719 1157 ts_tablet_manager.cc:1905] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.778097 2807 raft_consensus.cc:2743] T b56d2c0706c84195acceb97ef4985c14 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.778226 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31de9638669040db9674d3aa7df32103" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.782528 712 leader_election.cc:343] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.782635 712 leader_election.cc:304] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.783078 2807 raft_consensus.cc:2743] T 31de9638669040db9674d3aa7df32103 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:12.783843 854 leader_election.cc:343] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 72a6cf525cb44c428f7a39a25c4e62dc
I20250109 20:53:12.784765 854 leader_election.cc:304] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.785086 2818 raft_consensus.cc:2743] T 72a6cf525cb44c428f7a39a25c4e62dc P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.785728 1176 tablet_bootstrap.cc:492] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:12.785878 1176 ts_tablet_manager.cc:1397] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.120s user 0.003s sys 0.000s
I20250109 20:53:12.786841 1176 raft_consensus.cc:357] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.787084 1176 raft_consensus.cc:383] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.787163 1176 raft_consensus.cc:738] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.787487 1176 consensus_queue.cc:260] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.788383 1176 ts_tablet_manager.cc:1428] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.788959 1176 tablet_bootstrap.cc:492] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:12.789887 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet e54e9be016584fde911c4524969d0789
I20250109 20:53:12.790419 451 catalog_manager.cc:6596] ReplaceTablet: tablet e54e9be016584fde911c4524969d0789 deleted and replaced by tablet 18580b03f92f446d9a1cf06df2477b50
I20250109 20:53:12.790817 1176 tablet_bootstrap.cc:654] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.790755 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e82bee8a019c4cf4b23febe15b5c5f4e" } modify_external_catalogs: true
I20250109 20:53:12.790921 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e82bee8a019c4cf4b23febe15b5c5f4e" } modify_external_catalogs: true
I20250109 20:53:12.791774 2807 raft_consensus.cc:491] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.791927 2807 raft_consensus.cc:513] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.792637 2807 leader_election.cc:290] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.794474 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e54e9be016584fde911c4524969d0789 with delete_type TABLET_DATA_DELETED (replaced by tablet 18580b03f92f446d9a1cf06df2477b50) from {username='slave'} at 127.0.0.1:37640
I20250109 20:53:12.795195 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e54e9be016584fde911c4524969d0789 with delete_type TABLET_DATA_DELETED (replaced by tablet 18580b03f92f446d9a1cf06df2477b50) from {username='slave'} at 127.0.0.1:57642
W20250109 20:53:12.798020 713 leader_election.cc:343] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 12e97ff530df46ae80eb1b5615a55a3f
I20250109 20:53:12.799646 1158 tablet_bootstrap.cc:492] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.799763 1158 ts_tablet_manager.cc:1397] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.150s user 0.023s sys 0.031s
I20250109 20:53:12.803844 1158 raft_consensus.cc:357] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.804028 1158 raft_consensus.cc:383] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.804090 1158 raft_consensus.cc:738] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.804276 1158 consensus_queue.cc:260] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } }
I20250109 20:53:12.804855 1158 ts_tablet_manager.cc:1428] T 4f90c364ce844c9ea8eccb4ae23f8dce P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250109 20:53:12.805123 1158 tablet_bootstrap.cc:492] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.806702 1158 tablet_bootstrap.cc:654] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.807386 2818 raft_consensus.cc:491] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.810166 1157 ts_tablet_manager.cc:1918] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.810271 1157 log.cc:1198] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/f2c83e709a8d4e078e67af58ec51beac
I20250109 20:53:12.810590 1157 ts_tablet_manager.cc:1939] T f2c83e709a8d4e078e67af58ec51beac P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:12.807528 2818 raft_consensus.cc:513] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.811424 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1683) took 1286 ms. Trace:
I20250109 20:53:12.811617 2818 leader_election.cc:290] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.811517 1157 rpcz_store.cc:276] 0109 20:53:11.524817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:11.524890 (+ 73us) service_pool.cc:225] Handling call
0109 20:53:12.755284 (+1230394us) ts_tablet_manager.cc:1145] Deleting tablet f2c83e709a8d4e078e67af58ec51beac
0109 20:53:12.755296 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:12.806086 (+ 50790us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:12.810062 (+ 3976us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:12.811402 (+ 1340us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21333,"tablet-delete.queue_time_us":1223254}
W20250109 20:53:12.811933 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet e54e9be016584fde911c4524969d0789 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e54e9be016584fde911c4524969d0789 already in progress: creating tablet
I20250109 20:53:12.812000 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1679) took 1287 ms. Trace:
I20250109 20:53:12.812068 1157 rpcz_store.cc:276] 0109 20:53:11.524456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:11.524925 (+ 469us) service_pool.cc:225] Handling call
0109 20:53:12.811868 (+1286943us) ts_tablet_manager.cc:1145] Deleting tablet 88479a3f57654d84ad622af0782cce47
0109 20:53:12.811880 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:12.811983 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279708}
I20250109 20:53:12.812319 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet f2c83e709a8d4e078e67af58ec51beac (table default.Table_a7a796c2d1d047baa518202a6b0091b7 [id=4943d76200b14b51b15563815954e38e]) successfully deleted
I20250109 20:53:12.812361 1157 tablet_replica.cc:331] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:12.812574 1157 raft_consensus.cc:2238] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250109 20:53:12.812726 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet 88479a3f57654d84ad622af0782cce47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88479a3f57654d84ad622af0782cce47 already in progress: creating tablet
I20250109 20:53:12.813588 1176 tablet_bootstrap.cc:492] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:12.813726 1176 ts_tablet_manager.cc:1397] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20250109 20:53:12.813859 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
/tmp/dist-test-taskbTwFIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41583
--webserver_interface=127.0.104.60
--webserver_port=46069
--builtin_ntp_servers=127.0.104.20:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37941,127.0.104.61:39231,127.0.104.60:41583
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250109 20:53:12.814172 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e54e9be016584fde911c4524969d0789 with delete_type TABLET_DATA_DELETED (replaced by tablet 18580b03f92f446d9a1cf06df2477b50) from {username='slave'} at 127.0.0.1:52040
I20250109 20:53:12.814517 1176 raft_consensus.cc:357] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.814738 1176 raft_consensus.cc:383] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.814800 1176 raft_consensus.cc:738] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.815075 1176 consensus_queue.cc:260] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.816840 1176 ts_tablet_manager.cc:1428] T fd92f4c7abea454c8d68c1f5023d5727 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:53:12.818218 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e97ff530df46ae80eb1b5615a55a3f" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.818629 1176 tablet_bootstrap.cc:492] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
W20250109 20:53:12.819253 712 leader_election.cc:343] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.819351 712 leader_election.cc:304] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.819808 2920 raft_consensus.cc:2743] T 12e97ff530df46ae80eb1b5615a55a3f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:12.821010 712 consensus_peers.cc:487] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f -> Peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Couldn't send request to peer 44d473955919435cb679edd5f8afe2c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250109 20:53:12.821067 1176 tablet_bootstrap.cc:654] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.821851 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet e54e9be016584fde911c4524969d0789 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e54e9be016584fde911c4524969d0789 already in progress: creating tablet
I20250109 20:53:12.845707 1165 tablet_bootstrap.cc:492] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.845861 1165 ts_tablet_manager.cc:1397] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.015s
I20250109 20:53:12.846587 1165 raft_consensus.cc:357] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.846766 1165 raft_consensus.cc:383] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.846814 1165 raft_consensus.cc:738] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.847083 1165 consensus_queue.cc:260] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.847654 1165 ts_tablet_manager.cc:1428] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250109 20:53:12.847903 1165 tablet_bootstrap.cc:492] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.850661 1165 tablet_bootstrap.cc:654] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.852291 1157 raft_consensus.cc:2267] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:12.853603 1157 ts_tablet_manager.cc:1905] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:12.861501 1165 tablet_bootstrap.cc:492] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.862318 1165 ts_tablet_manager.cc:1397] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250109 20:53:12.862969 1165 raft_consensus.cc:357] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.863186 1165 raft_consensus.cc:383] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.863234 1165 raft_consensus.cc:738] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.863407 1165 consensus_queue.cc:260] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.863828 1165 ts_tablet_manager.cc:1428] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250109 20:53:12.864061 1165 tablet_bootstrap.cc:492] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.857141 2807 consensus_queue.cc:1035] T 74f4bdda16554d9d81de5ca5c5205f4c P 09833ea46b7141d58c3899dace69e49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250109 20:53:12.868897 1165 tablet_bootstrap.cc:654] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.872852 2912 raft_consensus.cc:491] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.872985 2912 raft_consensus.cc:513] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.873507 2912 leader_election.cc:290] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:12.873981 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617029392e994f44a60c71ba2e340c57" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
W20250109 20:53:12.874420 990 leader_election.cc:343] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.874501 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8328960890b142248073cec450e957d8: Sending DeleteTablet for 3 replicas of tablet f948f9c3f77d498a9043382d6d3ba72e
I20250109 20:53:12.874919 455 catalog_manager.cc:6596] ReplaceTablet: tablet f948f9c3f77d498a9043382d6d3ba72e deleted and replaced by tablet b21e9f37e2904c7293c11e81f8738b55
I20250109 20:53:12.874912 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617029392e994f44a60c71ba2e340c57" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.875362 989 leader_election.cc:343] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.875452 989 leader_election.cc:304] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.875838 2912 raft_consensus.cc:2743] T 617029392e994f44a60c71ba2e340c57 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250109 20:53:12.876422 854 leader_election.cc:343] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 1b50c91807734e86840e874886dc0749
W20250109 20:53:12.876658 854 leader_election.cc:343] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 1b50c91807734e86840e874886dc0749
I20250109 20:53:12.876801 854 leader_election.cc:304] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.877193 2818 raft_consensus.cc:2743] T 1b50c91807734e86840e874886dc0749 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.881824 1165 tablet_bootstrap.cc:492] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2: No bootstrap required, opened a new log
I20250109 20:53:12.881942 1165 ts_tablet_manager.cc:1397] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.002s
I20250109 20:53:12.882638 1165 raft_consensus.cc:357] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.882843 1165 raft_consensus.cc:383] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.882889 1165 raft_consensus.cc:738] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d473955919435cb679edd5f8afe2c2, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.883102 1165 consensus_queue.cc:260] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.883703 1165 ts_tablet_manager.cc:1428] T 3293f49190684b72a5c41031e621bd36 P 44d473955919435cb679edd5f8afe2c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250109 20:53:12.883967 1165 tablet_bootstrap.cc:492] T 3af9563f6efd47059caac7c86c41d43f P 44d473955919435cb679edd5f8afe2c2: Bootstrap starting.
I20250109 20:53:12.884459 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f948f9c3f77d498a9043382d6d3ba72e with delete_type TABLET_DATA_DELETED (replaced by tablet b21e9f37e2904c7293c11e81f8738b55) from {username='slave'} at 127.0.0.1:37640
W20250109 20:53:12.885038 441 catalog_manager.cc:4852] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): delete failed for tablet f948f9c3f77d498a9043382d6d3ba72e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f948f9c3f77d498a9043382d6d3ba72e already in progress: creating tablet
I20250109 20:53:12.885258 888 tablet_service.cc:1514] Processing DeleteTablet for tablet f948f9c3f77d498a9043382d6d3ba72e with delete_type TABLET_DATA_DELETED (replaced by tablet b21e9f37e2904c7293c11e81f8738b55) from {username='slave'} at 127.0.0.1:57642
I20250109 20:53:12.885465 752 tablet_service.cc:1514] Processing DeleteTablet for tablet f948f9c3f77d498a9043382d6d3ba72e with delete_type TABLET_DATA_DELETED (replaced by tablet b21e9f37e2904c7293c11e81f8738b55) from {username='slave'} at 127.0.0.1:52040
W20250109 20:53:12.885749 440 catalog_manager.cc:4852] TS 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): delete failed for tablet f948f9c3f77d498a9043382d6d3ba72e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f948f9c3f77d498a9043382d6d3ba72e already in progress: creating tablet
I20250109 20:53:12.894373 2807 raft_consensus.cc:491] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.894553 2807 raft_consensus.cc:513] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.895668 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df7405e93aa4718a62933ae56af9f42" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.895960 2807 leader_election.cc:290] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.895931 713 leader_election.cc:343] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 7df7405e93aa4718a62933ae56af9f42
W20250109 20:53:12.896149 712 leader_election.cc:343] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.896257 712 leader_election.cc:304] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.899843 2807 raft_consensus.cc:2743] T 7df7405e93aa4718a62933ae56af9f42 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.916013 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_9b490cee97c442edac131ba0a23a3e75" } new_table_name: "default.Table_074760b4e0714ed4b20a23088fad228f" modify_external_catalogs: true
I20250109 20:53:12.917620 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_b0b9646173584ad1b385b18ea22ad049" } new_table_name: "default.Table_813637c234f84337a024fa281bbbc721" modify_external_catalogs: true
I20250109 20:53:12.918706 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e6a87107149f421e93c2ef7e26400ef1" } modify_external_catalogs: true
I20250109 20:53:12.918869 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e6a87107149f421e93c2ef7e26400ef1" } modify_external_catalogs: true
I20250109 20:53:12.927166 1176 tablet_bootstrap.cc:492] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:12.927318 1176 ts_tablet_manager.cc:1397] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.109s user 0.005s sys 0.002s
I20250109 20:53:12.928047 1176 raft_consensus.cc:357] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.928222 1176 raft_consensus.cc:383] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.928268 1176 raft_consensus.cc:738] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.928467 1176 consensus_queue.cc:260] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.929046 1158 tablet_bootstrap.cc:492] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:12.929230 1158 ts_tablet_manager.cc:1397] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.124s user 0.017s sys 0.029s
I20250109 20:53:12.929864 1158 raft_consensus.cc:357] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.930107 1158 raft_consensus.cc:383] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:12.930205 1158 raft_consensus.cc:738] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:12.930459 1158 consensus_queue.cc:260] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.932063 1158 ts_tablet_manager.cc:1428] T f8eba3dcab764a98857604b63bf9dce1 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250109 20:53:12.932291 1176 ts_tablet_manager.cc:1428] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250109 20:53:12.933173 1158 tablet_bootstrap.cc:492] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:12.933251 1176 tablet_bootstrap.cc:492] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:12.934753 1158 tablet_bootstrap.cc:654] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.936353 2807 raft_consensus.cc:491] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.938058 2807 raft_consensus.cc:513] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.938131 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_692e5797e1314ab8ba6b6d1ac19e9cf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250109 20:53:12.941555 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c88f42c15d94099903178218bfac1e5" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:12.941634 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_696e8a1e0e3145bab03b3d71484d37cc" } new_table_name: "default.Table_59ccc53648c7446eab613aa2c810f9cf" modify_external_catalogs: true
W20250109 20:53:12.942060 712 leader_election.cc:343] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.942513 2807 leader_election.cc:290] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.942808 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4ed34f82b8c04d07b91df48899ffdfc2" } modify_external_catalogs: true
I20250109 20:53:12.942943 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_4ed34f82b8c04d07b91df48899ffdfc2" } modify_external_catalogs: true
W20250109 20:53:12.947448 713 leader_election.cc:343] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 9c88f42c15d94099903178218bfac1e5
I20250109 20:53:12.947548 713 leader_election.cc:304] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f; no voters: 44d473955919435cb679edd5f8afe2c2, 5fcc647ed479491b88e5ffacf2927a1a
W20250109 20:53:12.948258 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6600b4dff14949ddbd69a4d098be830d has no consensus state
W20250109 20:53:12.948961 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8926204e314f486dbdb7d0b14538b547 consensus state has no leader
I20250109 20:53:12.949247 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_e340970bd22c4332b51c18771b943166" } new_table_name: "default.Table_52d923ce3c3c428e831a66fb7bad961d" modify_external_catalogs: true
I20250109 20:53:12.948637 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7202bebfc9474ba2b0ad805191200c6f" } modify_external_catalogs: true
I20250109 20:53:12.950035 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_7202bebfc9474ba2b0ad805191200c6f" } modify_external_catalogs: true
W20250109 20:53:12.954154 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd799b72a7f445089655304d561c299d consensus state has no leader
I20250109 20:53:12.954418 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5bc556c741f74a51ac51116ee71ec9cd" } modify_external_catalogs: true
I20250109 20:53:12.954550 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_5bc556c741f74a51ac51116ee71ec9cd" } modify_external_catalogs: true
W20250109 20:53:12.955457 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_692e5797e1314ab8ba6b6d1ac19e9cf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.956820 2807 raft_consensus.cc:2743] T 9c88f42c15d94099903178218bfac1e5 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.961799 1165 tablet_bootstrap.cc:654] T 3af9563f6efd47059caac7c86c41d43f P 44d473955919435cb679edd5f8afe2c2: Neither blocks nor log segments found. Creating new log.
W20250109 20:53:12.964073 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36d28fe90bbf45178de37e52369aea22 consensus state has no leader
I20250109 20:53:12.964422 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_6d3cd554b3c04465adb8690b1ea507ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.964866 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d3cd554b3c04465adb8690b1ea507ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.965029 1176 tablet_bootstrap.cc:654] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:12.971272 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_a94d3e39950c4f63b7778efe4f3dd403"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.971715 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a94d3e39950c4f63b7778efe4f3dd403 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.972990 2807 raft_consensus.cc:491] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.973117 2807 raft_consensus.cc:513] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.974020 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50039da210df47ab8476e28387c8a802" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
W20250109 20:53:12.974572 712 leader_election.cc:343] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.975126 2807 leader_election.cc:290] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.975215 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50039da210df47ab8476e28387c8a802" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:12.975382 1050 raft_consensus.cc:2463] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 0.
I20250109 20:53:12.975780 713 leader_election.cc:304] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.976251 2807 raft_consensus.cc:2798] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250109 20:53:12.976538 2807 raft_consensus.cc:491] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250109 20:53:12.977113 2807 raft_consensus.cc:3054] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.980412 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_07af13c5c90841e986b40c45ad752f3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.980842 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07af13c5c90841e986b40c45ad752f3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250109 20:53:12.981019 2807 raft_consensus.cc:513] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.982095 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50039da210df47ab8476e28387c8a802" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2"
W20250109 20:53:12.982663 712 leader_election.cc:343] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:12.983218 2807 leader_election.cc:290] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Requested vote from peers 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:12.983287 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50039da210df47ab8476e28387c8a802" candidate_uuid: "09833ea46b7141d58c3899dace69e49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a"
I20250109 20:53:12.983383 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 2975f30fccea4d25b3001a0b5a91d8d4 from {username='slave'} at 127.0.0.1:33768
I20250109 20:53:12.983438 1050 raft_consensus.cc:3054] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Advancing to term 1
I20250109 20:53:12.984530 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33768:
table { table_name: "default.Table_03765feae81a4760b3e7064beec92515" } new_table_name: "default.Table_7ec54e9baa60446a9d4ba0c184693046" modify_external_catalogs: true
I20250109 20:53:12.985769 1050 raft_consensus.cc:2463] T 50039da210df47ab8476e28387c8a802 P 5fcc647ed479491b88e5ffacf2927a1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09833ea46b7141d58c3899dace69e49f in term 1.
I20250109 20:53:12.986335 713 leader_election.cc:304] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a; no voters: 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:12.986752 2807 raft_consensus.cc:2798] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Leader election won for term 1
I20250109 20:53:12.986910 2818 raft_consensus.cc:491] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:12.987044 2818 raft_consensus.cc:513] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.987186 2807 raft_consensus.cc:695] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Becoming Leader. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Running, Role: LEADER
I20250109 20:53:12.987452 2807 consensus_queue.cc:237] T 50039da210df47ab8476e28387c8a802 P 09833ea46b7141d58c3899dace69e49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:12.987627 2818 leader_election.cc:290] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:12.988688 854 leader_election.cc:343] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: dc235614171f4a61b75492ddeb3f0fe6
W20250109 20:53:12.988984 854 leader_election.cc:343] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: dc235614171f4a61b75492ddeb3f0fe6
I20250109 20:53:12.989080 854 leader_election.cc:304] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:12.989323 2818 raft_consensus.cc:2743] T dc235614171f4a61b75492ddeb3f0fe6 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:12.992296 1157 ts_tablet_manager.cc:1918] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250109 20:53:12.994326 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33768:
name: "default.Table_9130d502dc3e4eff8ae7d8bb4bbfff25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250109 20:53:12.994777 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9130d502dc3e4eff8ae7d8bb4bbfff25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250109 20:53:12.996197 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eaea4f239cce41a997496a1c94810ac9 consensus state has no leader
I20250109 20:53:13.004338 446 catalog_manager.cc:5618] TS 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949) does not have the latest schema for tablet 2f53cef0adbb47d4840e4c11a6d0c8ac (table default.Table_e3593cefd5e948bfb6e25e78caaff8de [id=c8701071e1d6496686497a84816c186c]). Expected version 1 got 0
I20250109 20:53:13.004633 2818 raft_consensus.cc:491] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:13.004812 2818 raft_consensus.cc:513] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.005553 2818 leader_election.cc:290] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
W20250109 20:53:13.007627 854 leader_election.cc:343] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Not found: Tablet not found: 829bec4f66c24180afd5271057a7d0e8
W20250109 20:53:13.007848 854 leader_election.cc:343] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949): Not found: Tablet not found: 829bec4f66c24180afd5271057a7d0e8
I20250109 20:53:13.007978 854 leader_election.cc:304] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:13.008634 2818 raft_consensus.cc:2743] T 829bec4f66c24180afd5271057a7d0e8 P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:13.016145 888 tablet_service.cc:1467] Processing CreateTablet for tablet 6600b4dff14949ddbd69a4d098be830d (DEFAULT_TABLE table=default.Table_074760b4e0714ed4b20a23088fad228f [id=48f60944edbb466e8f5aa8c609044bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.016559 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6600b4dff14949ddbd69a4d098be830d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.019685 1176 tablet_bootstrap.cc:492] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:13.019794 1176 ts_tablet_manager.cc:1397] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.000s
I20250109 20:53:13.020429 1176 raft_consensus.cc:357] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.020579 1176 raft_consensus.cc:383] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:13.020614 1176 raft_consensus.cc:738] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:13.020833 1176 consensus_queue.cc:260] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.020987 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 6600b4dff14949ddbd69a4d098be830d (DEFAULT_TABLE table=default.Table_074760b4e0714ed4b20a23088fad228f [id=48f60944edbb466e8f5aa8c609044bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.021338 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6600b4dff14949ddbd69a4d098be830d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.022045 1157 log.cc:1198] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: Deleting WAL directory at /tmp/dist-test-taskbTwFIA/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736455958930225-416-0/minicluster-data/ts-0/wal/wals/3af9563f6efd47059caac7c86c41d43f
I20250109 20:53:13.022122 1176 ts_tablet_manager.cc:1428] T 17940f311b0440a59a42cc57f3a0bc4c P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250109 20:53:13.022388 1157 ts_tablet_manager.cc:1939] T 3af9563f6efd47059caac7c86c41d43f P 09833ea46b7141d58c3899dace69e49f: Deleting consensus metadata
I20250109 20:53:13.023255 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1681) took 1498 ms. Trace:
I20250109 20:53:13.023388 1157 rpcz_store.cc:276] 0109 20:53:11.524630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:11.525045 (+ 415us) service_pool.cc:225] Handling call
0109 20:53:12.812316 (+1287271us) ts_tablet_manager.cc:1145] Deleting tablet 3af9563f6efd47059caac7c86c41d43f
0109 20:53:12.812325 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:12.911888 (+ 99563us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:12.992185 (+ 80297us) tablet_metadata.cc:641] Metadata flushed
0109 20:53:13.023232 (+ 31047us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280087}
I20250109 20:53:13.024019 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52040 (request call id 1685) took 1447 ms. Trace:
I20250109 20:53:13.024133 1157 rpcz_store.cc:276] 0109 20:53:11.576206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0109 20:53:11.576852 (+ 646us) service_pool.cc:225] Handling call
0109 20:53:13.023847 (+1446995us) ts_tablet_manager.cc:1145] Deleting tablet bfa3e55fc83243a3b4978e981fdadf8c
0109 20:53:13.023860 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0109 20:53:13.023981 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1446827}
I20250109 20:53:13.024538 1157 tablet_replica.cc:331] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f: stopping tablet replica
I20250109 20:53:13.024801 1157 raft_consensus.cc:2238] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 1 LEADER]: Raft consensus shutting down.
I20250109 20:53:13.025255 1157 raft_consensus.cc:2267] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250109 20:53:13.025563 1176 tablet_bootstrap.cc:492] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:13.027117 1176 tablet_bootstrap.cc:654] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:13.027849 441 catalog_manager.cc:4872] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): tablet 3af9563f6efd47059caac7c86c41d43f (table default.Table_c1a38a8dc6d24ae99667d26b2f0a44a3 [id=bb7d5ce383b74f8ea9adc434fd534973]) successfully deleted
W20250109 20:53:13.028050 441 catalog_manager.cc:4852] TS 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): delete failed for tablet bfa3e55fc83243a3b4978e981fdadf8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfa3e55fc83243a3b4978e981fdadf8c already in progress: creating tablet
I20250109 20:53:13.029150 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6600b4dff14949ddbd69a4d098be830d (DEFAULT_TABLE table=default.Table_074760b4e0714ed4b20a23088fad228f [id=48f60944edbb466e8f5aa8c609044bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.029469 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6600b4dff14949ddbd69a4d098be830d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.033017 740 tablet_service.cc:1467] Processing CreateTablet for tablet 5e90ac6b64f348f3905061012094208d (DEFAULT_TABLE table=default.Table_9581bb46aecf427eb75a633db2adda3a [id=a9ed31d745624c93aa3932310c0549ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.033301 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e90ac6b64f348f3905061012094208d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.037115 757 tablet_service.cc:1467] Processing CreateTablet for tablet 0d909c18bd6b4623a907375161850505 (DEFAULT_TABLE table=default.Table_fbcb20c87fa34c8cb9c40e7098d01395 [id=451f3c3affd74c68b3e4afdb3edabd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.037499 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d909c18bd6b4623a907375161850505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.038151 1157 ts_tablet_manager.cc:1905] T 3c5e9b31b27949c2948cb64b6124ee72 P 09833ea46b7141d58c3899dace69e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250109 20:53:13.044749 2912 raft_consensus.cc:491] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:13.044826 749 tablet_service.cc:1467] Processing CreateTablet for tablet d150f65284d44b7b91b91132e9d8f764 (DEFAULT_TABLE table=default.Table_bc61fb7c754b48f2868416a0158388f6 [id=0d053ae70dad46eb8616ac1369d46ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.044872 2912 raft_consensus.cc:513] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.045143 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d150f65284d44b7b91b91132e9d8f764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.045410 2912 leader_election.cc:290] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233)
I20250109 20:53:13.045886 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9382be89bcf1452ab33f3deede173d26" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:13.046012 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9382be89bcf1452ab33f3deede173d26" candidate_uuid: "5fcc647ed479491b88e5ffacf2927a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d473955919435cb679edd5f8afe2c2" is_pre_election: true
I20250109 20:53:13.046172 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 0d909c18bd6b4623a907375161850505 (DEFAULT_TABLE table=default.Table_fbcb20c87fa34c8cb9c40e7098d01395 [id=451f3c3affd74c68b3e4afdb3edabd1d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:13.046418 990 leader_election.cc:343] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:13.046514 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d909c18bd6b4623a907375161850505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.046489 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 5e90ac6b64f348f3905061012094208d (DEFAULT_TABLE table=default.Table_9581bb46aecf427eb75a633db2adda3a [id=a9ed31d745624c93aa3932310c0549ab]), partition=RANGE (key) PARTITION UNBOUNDED
W20250109 20:53:13.046514 989 leader_election.cc:343] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d473955919435cb679edd5f8afe2c2 (127.0.104.2:42233): Illegal state: must be running to vote when last-logged opid is not known
I20250109 20:53:13.046633 989 leader_election.cc:304] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fcc647ed479491b88e5ffacf2927a1a; no voters: 09833ea46b7141d58c3899dace69e49f, 44d473955919435cb679edd5f8afe2c2
I20250109 20:53:13.046773 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e90ac6b64f348f3905061012094208d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.047032 2912 raft_consensus.cc:2743] T 9382be89bcf1452ab33f3deede173d26 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250109 20:53:13.048976 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d150f65284d44b7b91b91132e9d8f764 (DEFAULT_TABLE table=default.Table_bc61fb7c754b48f2868416a0158388f6 [id=0d053ae70dad46eb8616ac1369d46ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.049268 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d150f65284d44b7b91b91132e9d8f764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.052943 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8e7a223be64047b8344bd40d085e83 (DEFAULT_TABLE table=default.Table_54fde017f7de4545a00254c29d41224d [id=169fee49680a42d5809ce855d76917b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.053212 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f8e7a223be64047b8344bd40d085e83. 1 dirs total, 0 dirs full, 0 dirs failed
W20250109 20:53:13.056905 971 tablet.cc:2367] T 2797e1df3d12476f89fb5155523c3935 P 44d473955919435cb679edd5f8afe2c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250109 20:53:13.057152 1158 tablet_bootstrap.cc:492] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:13.057336 1158 ts_tablet_manager.cc:1397] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.124s user 0.015s sys 0.017s
I20250109 20:53:13.057972 1158 raft_consensus.cc:357] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.057649 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8e7a223be64047b8344bd40d085e83 (DEFAULT_TABLE table=default.Table_54fde017f7de4545a00254c29d41224d [id=169fee49680a42d5809ce855d76917b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.058154 1158 raft_consensus.cc:383] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:13.058215 1158 raft_consensus.cc:738] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09833ea46b7141d58c3899dace69e49f, State: Initialized, Role: FOLLOWER
I20250109 20:53:13.058388 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f8e7a223be64047b8344bd40d085e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.058405 1158 consensus_queue.cc:260] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.058876 1158 ts_tablet_manager.cc:1428] T 37ee437300414eccba9926e824f72f27 P 09833ea46b7141d58c3899dace69e49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250109 20:53:13.059144 1158 tablet_bootstrap.cc:492] T faeec3dbcaff4de98e94357f20e7e8ad P 09833ea46b7141d58c3899dace69e49f: Bootstrap starting.
I20250109 20:53:13.060590 1158 tablet_bootstrap.cc:654] T faeec3dbcaff4de98e94357f20e7e8ad P 09833ea46b7141d58c3899dace69e49f: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:13.064196 897 tablet_service.cc:1467] Processing CreateTablet for tablet 32f1f8a0e4ab4d61a6a26c4f7053d193 (DEFAULT_TABLE table=default.Table_5c4a7abac8a742b697104214d095c2d1 [id=317aa9451b3846e5bc5e9be919af9765]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.064244 882 tablet_service.cc:1467] Processing CreateTablet for tablet 15918e20930e4ba1a71930be42898f52 (DEFAULT_TABLE table=default.Table_7c925d21560f4a888144a3c8bc75a2c2 [id=5333fae5b63440eb8d76cbcce41a1999]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.064561 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32f1f8a0e4ab4d61a6a26c4f7053d193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.064793 894 tablet_service.cc:1467] Processing CreateTablet for tablet f68906d0c9dc40dc95b00223944945ad (DEFAULT_TABLE table=default.Table_81d6c3babee64540bb908d606439d383 [id=8b0f0d38d25a439aa52e68171ed89d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.065181 891 tablet_service.cc:1467] Processing CreateTablet for tablet e06224406e5949fd8f9c48795cd573dd (DEFAULT_TABLE table=default.Table_d359b13e7edb4d83bbf14b911bc2f52d [id=457c66e4a07249d78d5550fa7b7e9763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.065531 880 tablet_service.cc:1467] Processing CreateTablet for tablet 5e90ac6b64f348f3905061012094208d (DEFAULT_TABLE table=default.Table_9581bb46aecf427eb75a633db2adda3a [id=a9ed31d745624c93aa3932310c0549ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.065798 892 tablet_service.cc:1467] Processing CreateTablet for tablet d150f65284d44b7b91b91132e9d8f764 (DEFAULT_TABLE table=default.Table_bc61fb7c754b48f2868416a0158388f6 [id=0d053ae70dad46eb8616ac1369d46ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.065919 1176 tablet_bootstrap.cc:492] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:13.066028 1176 ts_tablet_manager.cc:1397] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.009s
I20250109 20:53:13.066056 895 tablet_service.cc:1467] Processing CreateTablet for tablet 0d909c18bd6b4623a907375161850505 (DEFAULT_TABLE table=default.Table_fbcb20c87fa34c8cb9c40e7098d01395 [id=451f3c3affd74c68b3e4afdb3edabd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.066385 898 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8e7a223be64047b8344bd40d085e83 (DEFAULT_TABLE table=default.Table_54fde017f7de4545a00254c29d41224d [id=169fee49680a42d5809ce855d76917b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.066732 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8762d25790c64f53b0590680a8c3f76b (DEFAULT_TABLE table=default.Table_e5b00a1d5f624d3fb6ff7d93ccf66815 [id=4a899737e96f4e04b64849e01e05d673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.066766 1176 raft_consensus.cc:357] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.066938 1176 raft_consensus.cc:383] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:13.066983 1176 raft_consensus.cc:738] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:13.067028 884 tablet_service.cc:1467] Processing CreateTablet for tablet b05271c38bfc400ebea09c18ec534d64 (DEFAULT_TABLE table=default.Table_c4bf8dafa0e24de5a186f5cee909662f [id=4b5a7e48c6644751aaa41261aa9862f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.067190 1176 consensus_queue.cc:260] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.067342 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8956a93ce64351b6d1985766720ba5 (DEFAULT_TABLE table=default.Table_2c4291be2f56426cafb273138c51dece [id=d556febb271c4538b013806c033b39cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.067646 889 tablet_service.cc:1467] Processing CreateTablet for tablet 81aa59fa65454cb7a4bbcbdc5ce3a2f9 (DEFAULT_TABLE table=default.Table_3f395e4c8c84426aa4888bd424686c0d [id=1bb6082176174a3189c20bd13a448024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.067919 886 tablet_service.cc:1467] Processing CreateTablet for tablet ebd8ffc36c0e48049fb9ae474a40a91f (DEFAULT_TABLE table=default.Table_5ba80d2ae9b14d0b94259012c5686f28 [id=947f6194b58e43e2a0e948d24f44621e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.068176 887 tablet_service.cc:1467] Processing CreateTablet for tablet bb994f8b099e43639c40d3fc304fe83a (DEFAULT_TABLE table=default.Table_a2156eacd7fd4fcba8e1649d9a83ad81 [id=9b59eb542903469c81bb7581edd4c872]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.068429 883 tablet_service.cc:1467] Processing CreateTablet for tablet ca2b973bee6c4ab18dcc431f52701a5d (DEFAULT_TABLE table=default.Table_7318b2f52d6846e6a5c719f130fae1c3 [id=54eed2b32deb4c2fb87885e35c2c7b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.068696 896 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac0dbb5e1dc41a5b6e50fe38ce97bf0 (DEFAULT_TABLE table=default.Table_95fcb9f283734637832d08ce71e8277e [id=b8ea074256ec4c40bfce44bed8541a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.068873 1176 ts_tablet_manager.cc:1428] T 0e9041a2e91f4753aa561965cc0e6280 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250109 20:53:13.068946 881 tablet_service.cc:1467] Processing CreateTablet for tablet b246bbc150c946cda2531d460c02fbc0 (DEFAULT_TABLE table=default.Table_7afd8872782446a39a98ace49db8ebb6 [id=95e0bcdf8eb14c4dbf436cca9ce4d433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.069037 893 tablet_service.cc:1467] Processing CreateTablet for tablet e201b4bbcc89422eb40c8eaa46942916 (DEFAULT_TABLE table=default.Table_229ce97b77fe4d589363707c89b17ef7 [id=fec249f091cb44a298e7bc5e93c76493]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.069231 899 tablet_service.cc:1467] Processing CreateTablet for tablet 1353f486b3434b578eca9abb6e799edb (DEFAULT_TABLE table=default.Table_741be8472805423ba5df0e5dc57e2bcc [id=ffce18e9c72a42c3bc3244168bf87413]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.069511 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68906d0c9dc40dc95b00223944945ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.070294 1176 tablet_bootstrap.cc:492] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:13.071700 1176 tablet_bootstrap.cc:654] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:13.072901 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06224406e5949fd8f9c48795cd573dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.075600 1176 tablet_bootstrap.cc:492] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:13.075711 1176 ts_tablet_manager.cc:1397] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250109 20:53:13.076481 1176 raft_consensus.cc:357] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.076679 1176 raft_consensus.cc:383] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:13.076766 1176 raft_consensus.cc:738] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:13.078727 1176 consensus_queue.cc:260] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.078914 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d150f65284d44b7b91b91132e9d8f764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.079257 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e90ac6b64f348f3905061012094208d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.079416 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15918e20930e4ba1a71930be42898f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.079545 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81aa59fa65454cb7a4bbcbdc5ce3a2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.079668 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b246bbc150c946cda2531d460c02fbc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.080199 1176 ts_tablet_manager.cc:1428] T 7a1d5b0210134ce1a33b5e7603998593 P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250109 20:53:13.081559 1176 tablet_bootstrap.cc:492] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:13.082157 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e201b4bbcc89422eb40c8eaa46942916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.082973 1176 tablet_bootstrap.cc:654] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:13.084290 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb994f8b099e43639c40d3fc304fe83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.084808 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1353f486b3434b578eca9abb6e799edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.088687 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebd8ffc36c0e48049fb9ae474a40a91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.088714 1176 tablet_bootstrap.cc:492] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a: No bootstrap required, opened a new log
I20250109 20:53:13.088877 1176 ts_tablet_manager.cc:1397] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250109 20:53:13.089522 1176 raft_consensus.cc:357] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.089684 1176 raft_consensus.cc:383] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250109 20:53:13.089727 1176 raft_consensus.cc:738] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcc647ed479491b88e5ffacf2927a1a, State: Initialized, Role: FOLLOWER
I20250109 20:53:13.089903 1176 consensus_queue.cc:260] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.092816 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b05271c38bfc400ebea09c18ec534d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.093080 894 tablet_service.cc:1467] Processing CreateTablet for tablet 3c64b94db1404804a9ca1a4f52aff57f (DEFAULT_TABLE table=default.Table_8f3c8dbeed7645a5849426fa972200c5 [id=9757ed12a3c840d0a8de7324c55cace8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.093096 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d909c18bd6b4623a907375161850505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.093330 1176 ts_tablet_manager.cc:1428] T 39f0cc2be25940d08fad42a8c8d8a07b P 5fcc647ed479491b88e5ffacf2927a1a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250109 20:53:13.093489 2818 raft_consensus.cc:491] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250109 20:53:13.093504 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2b973bee6c4ab18dcc431f52701a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.093605 2818 raft_consensus.cc:513] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09833ea46b7141d58c3899dace69e49f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35465 } } peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42233 } } peers { permanent_uuid: "5fcc647ed479491b88e5ffacf2927a1a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36949 } }
I20250109 20:53:13.094183 2818 leader_election.cc:290] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09833ea46b7141d58c3899dace69e49f (127.0.104.1:35465), 5fcc647ed479491b88e5ffacf2927a1a (127.0.104.3:36949)
I20250109 20:53:13.094444 1176 tablet_bootstrap.cc:492] T 687db3e734af47339c2546970a71a539 P 5fcc647ed479491b88e5ffacf2927a1a: Bootstrap starting.
I20250109 20:53:13.095019 1050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcc647ed479491b88e5ffacf2927a1a" is_pre_election: true
I20250109 20:53:13.095938 1176 tablet_bootstrap.cc:654] T 687db3e734af47339c2546970a71a539 P 5fcc647ed479491b88e5ffacf2927a1a: Neither blocks nor log segments found. Creating new log.
I20250109 20:53:13.096858 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8762d25790c64f53b0590680a8c3f76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.098243 881 tablet_service.cc:1467] Processing CreateTablet for tablet fef0cade8d3143eb987a866e8746fce2 (DEFAULT_TABLE table=default.Table_96ef863aeef94bf5950c9be167dde185 [id=66ffa19a6d6f4692b856a41ae053671a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250109 20:53:13.098508 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fef0cade8d3143eb987a866e8746fce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.099822 778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee208bad76734a8fa1ada438bdbbe48c" candidate_uuid: "44d473955919435cb679edd5f8afe2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09833ea46b7141d58c3899dace69e49f" is_pre_election: true
I20250109 20:53:13.100523 854 leader_election.cc:304] T ee208bad76734a8fa1ada438bdbbe48c P 44d473955919435cb679edd5f8afe2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d473955919435cb679edd5f8afe2c2; no voters: 09833ea46b7141d58c3899dace69e49f, 5fcc647ed479491b88e5ffacf2927a1a
I20250109 20:53:13.101298 1158 tablet_bootstrap.cc:492] T faeec3dbcaff4de98e94357f20e7e8ad P 09833ea46b7141d58c3899dace69e49f: No bootstrap required, opened a new log
I20250109 20:53:13.101320 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f8e7a223be64047b8344bd40d085e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250109 20:53:13.101424 1158 ts_tablet_manager.cc:1397] T faeec3dbcaff4de98e94357f20e7e8ad P 09833ea46b7141d58c3899dace69e49f: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.013s
I20250109 20:53:13.102082 1158 raft_consensus.cc:357] T faeec3dbcaff4de98e94357f20e7e8ad P 09833ea46b7141d58c3899dace69e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d473955919435cb679edd5f8afe2c2" member_type: VOTER last_known_addr { host: "127.0.104.2"